Consume from Kafka and stream to table

info

For general concepts around streaming in Superblocks, see Streaming Applications.

This guide explains how to consume data from a Kafka based streaming platform and display that data live in a Superblocks Table component. We'll use Kafka in this example, but the same steps can be applied using the Confluent or Redpanda integrations.

1. Create a temporary frontend variable with a default value of an empty array. Here, we've named the variable orders.

2. Add a Table component and set its Table Data property to the value of the frontend variable, {{orders.value}}.

3. Create a backend API with a single Stream block. Here we've named the API getOrders and the Stream block orderStream.

4. Add your Kafka based integration to the Trigger section of the Stream block. Here, we've named the step consumeOrder. Configure the integration step as follows:
   • Set Action to "Consume"
   • Set Topic to your topic name
   • Set Consume From to your desired consumer settings (Beginning, Offset, or Latest)

5. Add a JavaScript step to the Process section of the Stream block. Here, we've named the step transformOrder and added code to return just the flattened order data we care about.

Access the message for the current iteration of the stream with message.value. Full code snippet below.

function flattenObject(obj, parentKey = '', separator = '_') {
    const flattened = {};

    for (const key in obj) {
        if (obj.hasOwnProperty(key)) {
        const newKey = parentKey ? parentKey + separator + key : key;

        if (typeof obj[key] === 'object' && !Array.isArray(obj[key])) {
            // Recursively call the function for nested objects
            Object.assign(flattened, flattenObject(obj[key], newKey, separator));
        } else {
            flattened[newKey] = obj[key];
        }
        }
    }
    return flattened;
    }

    let order = message.value.value;
    return  flattenObject(order);

Test running the API shows the data returned for each iteration of the stream is a flattened JSON object with that order's information.

6. Configure the response block of the API.

   • Set Response type to "Stream"
   • Add an onMessage event handler that sets the previously created frontend variable with the Set Frontend Variable action. Since the Table component requires an array of objects, we've set the frontend variable to an array that prepends the message returned from the stream (accessible through message.value) to the latest value of the frontend variable using JavaScript spread syntax.

   {{[message.value, ...<TEMPVAR>.value]}}
   

7. Run the API in edit mode to see orders streamed to the table.