Retrieve entire data from paginated API recursively.

In this post i am going to show to retrieve data from a paginated API using recursive method.

To explain the functionality, I am going to use a dummy REST API which i created in the mockapi.io. Currently the dataset has 91 records. By default it returns 91 records. In realtime, the dataset may be huge of more than 10000's data items. It's not recommended to query the entire dataset in a single api as it burdens the server performance and also place a heavy load on the client side as well. The recommended approach is to paginate the results when you are showing in the client side. That's the reason most of the API's provides some additional filters in the api to return only the records which the user is interested instead of returning the entire datastore.

Here some of the common filters used by the API's.

1.offset, limit : "limit" determines how many records to retrieve starting from the "offset"
2.page, limit : "page" determines which data the server needs to return based on number of "limit" -> records per page .

Please refer the respective API developer manual for more details on how to retrieve paginated results.

Sample API End Points :

Without Pagination :
https://5b5cb0546a725000148a67ab.mockapi.io/api/v1/users

With Pagination :
https://5b5cb0546a725000148a67ab.mockapi.io/api/v1/users?page=4&limit=2

The above API endpoints are good for a normal use case. But they are requirements in which you need to retrieve all the data from the API and the API is designed to return only 50 records per call. In this scenario, we can create a recursive paginated API call and capture the entire dataset.

Remember,If you are not handling this properly it may end up with infinite looping and your browser may hang. In this example i have limited 20 items per page.

const limitPerPage=20;
const apiUrl="https://5b5cb0546a725000148a67ab.mockapi.io/api/v1/users";

const getUsers = async function(pageNo = 1) {

let actualUrl=apiUrl + `?page=${pageNo}&limit=${limitPerPage}`;
var apiResults=await fetch(actualUrl)
.then(resp=>{
return resp.json();
});

return apiResults;

}

const getEntireUserList = async function(pageNo = 1) {
  const results = await getUsers(pageNo);
  console.log("Retreiving data from API for page : " + pageNo);
  if (results.length>0) {
    return results.concat(await getEntireUserList(pageNo+1));
  } else {
    return results;
  }
};


(async ()=>{

    const entireList=await getEntireUserList();
    console.log(entireList);

})();

I have used the browser developer console to show the output. Please see this gif http://recordit.co/p3FkQS7QpJ

Thanks for reading & happy coding..