Fetch paginated data from Airtable's API

This guide explains how to use a Loop block and a Variables block to iterate across paginated data in a REST API. This example uses Airtable’s API, so steps may need to be adjusted to match the data structure for other APIs.

Before Getting Started

• Be sure to set up the Airtable Integration in your account.

Motivation for using a loop block

Airtable's List Records endpoint returns paginated data, 100 records per page. To fetch the next page, an offset parameter is provided which can be included to get the next page of records. Thus, in order to get more than 100 records from Airtable's API, you'll need to iterate across the API until all records are pulled using a Loop block and passing in the offset parameter from each previous call. Follow the guide below for step-by-step instructions.

List Records in a loop

1. Create a new backend API named fetchAirtableRecords with a Variables block
2. Add 3 variables: offsetAPI with a value of true, nextPage with a value of null, and outputRecords with a value of []
   
3. Add a Loop block that runs while offsetAPI.value is true
   
4. Inside the Loop block, add a step that uses your Airtable integration
   • Select List Records from the Action menu
   • Add your baseId and tableIdOrName to the URL path
   • Add a query parameter with a key offset and value {{nextPage.value}}
     
5. Click into the 3 dots menu on the Variables1 block and click Copy block, then click on the Airtable block's 3 dots menu and click Paste block below
   • Update the 3 variables:
     • offsetAPI with a value of Step1.output.offset ? true : false
     • nextPage with a value of Step1.output.offset
     • outputRecords with a value of

     [...outputRecords.value, ...Step1.output.records.map((record) => {   const { id, fields } = record;   return { id, ...fields }; })]
   • offsetAPI now checks if the result of the List Records call contains an offset parameter and returns true or false accordingly. This allows the while loop to continue until the offset parameter stops being returned, indicating that there are no more pages
   • nextPage now updates to be the offset parameter to pass into the subsequent List Records calls via the query parameter
   • outputRecords flattens and appends the data returned by the API structuring the data to be visualized using the Table component
     
6. Outside the Loop block, create a Return block that returns outputRecords.value
   
7. Drag a Table component to the canvas and update the Table Data to {{fetchAirtableRecords.response}}
8. Run the API and watch the Table component visualize your Airtable data!

Update Airtable records

1. Create a backend API called updateRecords with an Airtable step
   • Select Update multiple records from the Action menu
   • Add your baseId and tableIdOrName to the URL path
2. Update the JSON body to the following:

   {
   "records": {{Table1.editedRows.updatedRows.map((o) => {
     const { id, ...fields } = o; // Destructure and remove id from fields
     return {
       fields,
       id, // Include id as a separate unique identifier
     };
   })}}
   }
   • This restructures the data that was flattened in Step 5 above and uses id as an identifier for the table
3. In the Table component, update columns to editable that will be updated in Airtable
4. In the Table's onSaveChanges event handler, run the updateRecords API and then run the fetchAirtableRecords API
   
5. Now, you can edit the columns that were set to editable, click the Save Change button and the underlying Airtable base will be updated and the data will be re-fetched