Skip to main content

Fetch paginated data from Airtable's API

This guide explains how to use a Loop block and a Variables block to iterate across paginated data in a REST API. This example uses Airtable’s API, so steps may need to be adjusted to match the data structure for other APIs.

Before Getting Started

Motivation for using a loop block

Airtable's List Records endpoint returns paginated data, 100 records per page. To fetch the next page, an offset parameter is provided which can be included to get the next page of records. Thus, in order to get more than 100 records from Airtable's API, you'll need to iterate across the API until all records are pulled using a Loop block and passing in the offset parameter from each previous call. Follow the guide below for step-by-step instructions.

List Records in a loop

  1. Create a new backend API named fetchAirtableRecords with a Variables block
  2. Add 3 variables: offsetAPI with a value of true, nextPage with a value of null, and outputRecords with a value of []
Create a backend api with variable block
  1. Add a Loop block that runs while offsetAPI.value is true
Add a while loop block
  1. Inside the Loop block, add a step that uses your Airtable integration
    • Select List Records from the Action menu
    • Add your baseId and tableIdOrName to the URL path
    • Add a query parameter with a key offset and value {{nextPage.value}}
Add the airtable step to list records
  1. Click into the 3 dots menu on the Variables1 block and click Copy block, then click on the Airtable block's 3 dots menu and click Paste block below
    • Update the 3 variables:

      • offsetAPI with a value of Step1.output.offset ? true : false
      • nextPage with a value of Step1.output.offset
      • outputRecords with a value of
      [...outputRecords.value, ...Step1.output.records.map((record) => {   const { id, fields } = record;   return { id, ...fields }; })]
    • offsetAPI now checks if the result of the List Records call contains an offset parameter and returns true or false accordingly. This allows the while loop to continue until the offset parameter stops being returned, indicating that there are no more pages

    • nextPage now updates to be the offset parameter to pass into the subsequent List Records calls via the query parameter

    • outputRecords flattens and appends the data returned by the API structuring the data to be visualized using the Table component

Update the variables block after each Airtable call
  1. Outside the Loop block, create a Return block that returns outputRecords.value
Add a return block to return the data to the frontend
  1. Drag a Table component to the canvas and update the Table Data to {{fetchAirtableRecords.response}}
  2. Run the API and watch the Table component visualize your Airtable data!

Update Airtable records

  1. Create a backend API called updateRecords with an Airtable step
    • Select Update multiple records from the Action menu
    • Add your baseId and tableIdOrName to the URL path
  2. Update the JSON body to the following:
{
"records": {{Table1.editedRows.updatedRows.map((o) => {
const { id, ...fields } = o; // Destructure and remove id from fields
return {
fields,
id, // Include id as a separate unique identifier
};
})}}
}
  • This restructures the data that was flattened in Step 5 above and uses id as an identifier for the table
  1. In the Table component, update columns to editable that will be updated in Airtable
  2. In the Table's onSaveChanges event handler, run the updateRecords API and then run the fetchAirtableRecords API
Run the updateRecords and fetchAirtableRecords APIs on save changes
  1. Now, you can edit the columns that were set to editable, click the Save Change button and the underlying Airtable base will be updated and the data will be re-fetched