Leveraging JavaScript to Bulk-Fetch Data from Airtable

If you are here, that either means you saw the title of this article and since you have been curious about how to solve fetching data in bulk from Airtable, you immediately clicked on the article, or you are a curious cat who loves reading about "bulk fetching data". Well, just in case you are part of the latter category, I will get started by talking about the wonderful platform called Airtable.

Like its name implies, Airtable is a table that floats on air, okay seriously, Airtable is a popular platform among developers for organizing data. Its intuitive drag-and-drop interface and powerful customization options make it easy to create beautiful, functional databases and spreadsheets. Plus, developers can easily link records across multiple tables to access all the data they need in one place. With its flexibility, developers can quickly and easily customize the layout, fields, and even the formulas used to calculate data. This makes Airtable great for storing and fetching data, allowing developers to easily organize tasks, track projects, and manage customer relationships.

If you are a frontend developer like me, and you don't particularly know how to write backend code then Airtable can be very helpful for creating a no-code backend for simple applications that involves just basic CRUD operations.

While Airtable is very easy to use, recently I discovered that you are only able to fetch 100 data at once and to get more data, you'd have to make another API call, so your paginated page is only restricted to 100 data at once. In my case, I really needed to get all the data all at once from the backend and that made me search extensively to discover how to bulk fetch data. So here I am, sharing this little gem so you don't have to do all the searching I did.

Well, let's get to the coding part. To bulk fetch data at once, you'd still have to get 100 data for one API call you make, then use recursion to call the function again till you have nothing left in your table. This method is not advisable if your data is more than 2000 records because you will end up hitting the endpoint so many times and your loading time will be too long, so you might want to consider other options.

Check out this little function to achieve the recursion functionality below.

 function getData({ offset, result = [] }) {
  let url;
  if (offset) {
    url = `https://api.airtable.com/v0/dataTable?pageSize=${items}&offset=${offset}`;
  } else
    url = `https://api.airtable.com/v0/dataTable?pageSize=${items}`;
  const response = await fetch(url, {
    method: "GET",
    headers: {
      Authorization: `Bearer 
${process.env.REACT_APP_AIRTABLE_API_KEY}`,
    },
  });
  const data = await response.json();
  result = result.concat(data);

  if (!response.ok) {
    throw new Error(data.detail || "Something went wrong.");
  }
  if (!data.offset) {
    console.log(result);
    return result;
  } else {
    console.log(result);
    return getData({ offset: data.offset, result });
  }
}

In this function we are accepting an object that contains an offset value and an array of result, which is initialized as empty. The offset is a string that is usually returned by the Airtable endpoint when a request is made for data that is more than 100 records. On our the first API call, the offset is false because we have not hit the endpoint yet.
We will use a conditional statement to check if this offset value is present, if it is, we will append it to the url.
The next step involves calling the endpoint using any method you would like, you can use inbuilt fetch method, axios or even react query, just ensure that the fetch method returns the correct data from the endpoint.

We will then add the data gotten from the endpoint to the currently empty result array.

The next step is to use another conditional statement to check if there is an offset value returned with the endpoint, if there is an offset value, it means that there are more records to be fetched, so we will call the getData({ offset: data.offset, result })
function again with the new offset value. If there is no offset returned then we will just return the result array with all the concatenated values.
There you have it, we were able to fetch all the values from Airtable using Javascript recursion method.

Byeeee!!!👋👋