Get All Paginated Data using RxJS Expand Operator

It’s common for an API to paginate the data it returns, instead of returning all of it. What if you had a scenario where you needed to get ALL paginated data at once? Let’s learn how with the RxJS expand operator!

Paginated API Example

Let’s look at an example of an API that returns paginated data. If we wanted to get a list of characters (the first 10 characters) from the Star Wars API, the endpoint for that would be:

https://swapi.dev/api/people/

The returned data follows. Notice that the next property provides the next api call to get data for page 2.

{
  count: 82,
  next: 'https://swapi.dev/api/people/?page=2',
  previous: null,
  results: []
}

The count tells us the total number of characters is 82. So if each page only returns 10, then there are 9 pages total. If we called for page 9, the returned data follows. Notice that the next property value is now null. That means there are no more pages.

{
  count: 82,
  next: null,
  previous: 'https://swapi.dev/api/people/?page=8',
  results: []
}

RxJS Expand Operator

If we needed to get all paginated data, we’d need to do a few things:

1. Make the API request for the initial page of data.
2. Within the returned data, check the next property to see if there’s a next page:
   • If the next page exists, make the API request for the next page.
   • If the next page does not exist, we’ve reached the end, stop calling the API.
3. While each page of data is returned, build up an array with all the entries.

Let’s start with the initial API request.

this.httpSvc.get('https://swapi.dev/api/people').pipe(

);

Within the pipe, let’s add the RxJS expand operator. This will give us the ability to continue calling each page until we’re done. It will receive the response from the API request. Let’s check the next property. If it exists, make an API request using the value for the next page. If it does not exist, call the RxJS EMPTY constant, which emits nothing and immediately completes.

this.httpSvc.get('https://swapi.dev/api/people').pipe(
  expand(response => response.next ? this.httpSvc.get(response.next) : EMPTY)
);

RxJS Reduce Operator

Right after the expand operator, let’s add the RxJS reduce operator. This will give us the ability to build up an array of all the entries. It will receive an accumulator, which starts as an empty array, as well as the data returned from the current API request. We want to concatenate the accumulator with the results property from the API request, which is also an array. When all is said and done, we’ll have an array of ALL paginated data!

this.httpSvc.get('https://swapi.dev/api/people').pipe(
  expand(response => response.next ? this.httpSvc.get(response.next) : EMPTY),
  reduce((acc, current) => acc.concat(current.results), [])
);

RxJS Delay Operator

Be aware that APIs may have restrictions in place on how often you can call them within a short time frame. If that’s the case, you can include a delay on the API request for subsequent pages. The following example delays the results for 1000 milliseconds (1 second) using the RxJS delay operator.

this.httpSvc.get(response.next).pipe(
  delay(1000)
)

Visit our website at https://nightwolf.dev and follow us on Twitter!