Overview

Streaming allows you to build real-time applications in Superblocks. Create live views with Kafka consumers, streaming chatbots with OpenAI, and more.

With Streaming in Superblocks, you can:

1. Connect streaming integrations
2. Read from a streaming integration in backend APIs
3. Stream data to the client with the Stream API response type and onMessage event handler
4. Write to a streaming integration in backend APIs

Connect streaming integrations

To get started with Streaming, set up any of following streaming integrations:

• Kafka or Kafka-compatible integrations like Confluent and Redpanda
• OpenAI

Read from a streaming integration

To read data from a streaming integration:

1. Add a Stream block in a backend API
2. Within the Stream block, add and configure your streaming integration step inside the Trigger section
3. Optionally, apply custom processing of that message inside the Process section

info

For Kafka-based integrations, the Stream block is intended only for Consume actions. Produce actions should be configured outside of a Stream block.

For more details on the Stream block, see Control Blocks.

Stream data to the client

Messages emitted from streaming APIs are sent to the client. In Applications, the client is the Application UI. In Workflows, the client is the machine making the request.

Sending messages can be accomplished in different ways depending on the context.

1. When reading from a streaming integration using the Stream block:

   • Send messages automatically by enabling the Stream block's Automatically send messages to client setting (default).
   • Alternatively, disable automatic sending and send messages manually using the Send block (e.g. to send messages in batches).

2. When running any backend API:

   • Use the Send block independently to send any backend data to the client. This is useful for streaming incremental results from APIs to the frontend outside of the context of streaming integrations.

In the case of streaming Workflows invoked using HTTP/2, these messages will be streamed back to the client machine making the request.

To use of any these features for sending data to the frontend of an Application, see the next sections for configuring the Stream API response type and onMessage event handler.

Stream API response type

To stream messages from the backend to the frontend of the Application, change the API response type to Stream in the API response block.

Unlike standard synchronous APIs, streaming APIs include an additional event handler, onMessage, that is triggered for each message sent to the frontend. The next section covers how to use this event handler to update UI components with data from the stream.

info

Streaming APIs do not have a response variable for referencing a final output (i.e., there is no API_NAME.response object).

onMessage event handler

As messages are sent from the backend to the frontend, they trigger the onMessage event handler. This event handler receives a message parameter which contains the current message.

To use these messages in your App UI, you'll need to store the data in a frontend state variable, and then bind your component to this state variable. To do so:

1. Create a temporary state variable
2. Update the state variable in the streaming API’s onMessage event handler
3. Bind the data property of a component to the state variable

Example

Imagine we want to build a list of messages from a streaming API to be displayed in a table.

1. Here we've created a state variable called items and initialized it to an empty array.
2. Each time a message is consumed and processed in the streaming API, the onMessage event handler updates the items state variable, adding the message's value to the front of the list using JavaScript spread syntax:

    {{[message.value, ...items.value]}}
3. The Table component is bound to the state variable's value via {{items.value}}.

When the API runs, each message consumed is streamed to the onMessage event handler which updates the state variable, resulting in a live feed of items in the Table.

Write to a streaming integration

To send data to write-compatible streaming integrations (e.g., producing messages in a Kafka-based service), add the integration step to a backend API, set the action to Produce, and configure the messages to write.

Additional resources

Learn more about how to leverage Streaming for your internal tools here:

• Technical Guide: Create a real-time chatbot with streaming and OpenAI
• Technical Guide: Consume from Kafka and stream to Table