Introduction
When working with large datasets, various techniques are employed to optimize queries, enhance user experience, and improve performance. One of those techniques is Pagination. Pagination involves dividing large datasets into smaller, manageable subsets.
Pagination can be implemented in two ways:
- Client-Side Pagination: In client-side pagination, the client application queries the entire dataset from the server, and pagination logic is handled in the browser. Client-side pagination is easy to implement for small datasets. Client-side pagination also prevents additional queries to the database.
- Server-Side Pagination: In server-side pagination, the server returns a subset of the data at a time when queried. Server-side pagination reduces data load on the client and improves performance for large datasets by only fetching necessary data.
In this beginner-friendly tutorial, you’ll implement server-side pagination in Express.js and MongoDB. You’ll create a collection for storing movies in MongoDB and an API endpoint that retrieves data from a MongoDB collection. We’ll cover two methods in this tutorial:
- Basic pagination implemented without the use of an external library.
- Pagination using the
mongoose-paginate-v2library.
Prerequisites
To follow along with this tutorial, you'll need:
- A code editor like Visual Studio Code or any code editor of your choice.
- An API client such as Postman for testing your API endpoints.
- Node.js installed on your computer.
- Git is installed on your computer.
- A MongoDB instance either locally or on the web using MongoDB Atlas.
- Basic knowledge of Express.js.
Boilerplate Setup
To keep the article concise, I have created a boilerplate code that can be accessed here on GitHub. The boilerplate contains the basic setup for an Express.js application.
To get started with the tutorial:
- Clone the repository: Firstly, clone the repository from GitHub using the following command: ``` bash git clone https://github.com/michaelikoko/Express-Pagination.git
2. **Navigate to the project directory**: The `starter` directory contains the boilerplate code. Navigate to the `starter` using the following command:
```
bash
cd Express-Pagination/starter
The `starter` directory should have the following folder structure:
- Install dependencies: Install the needed node dependencies, using any package manager you choose. For NPM run the following command: ``` bash npm install
4. **Connect to MongoDB**: Ensure you have a MongoDB instance running locally or on MongoDB Atlas. Create a `.env` file in the current directory, and provide the application port and database, as shown below:
```
dockerfile
PORT=5090
MONGODB_URI=mongodb+srv://<username>:<password>@cluster1.menjafx.mongodb.net/?retryWrites=true&w=majority
-
Create the Movie Schema: In the file
models/Movie.js, define the Movie Schema:javascript const mongoose = require("mongoose"); const movieSchema = new mongoose.Schema( { title: { type: String, required: true, minLength: [2, "Movie title length is too short"], maxLength: [100, "Movie title length is too long"], trim: true, }, director: { type: String, required: true, }, genre: { type: String, required: true, }, releaseYear: { type: Date, }, }, { timestamps: true } ); module.exports = mongoose.model("Movie", movieSchema);
6. **Populate the database**: To make testing easier, populate the database with mock data contained in `MOCK_DATA.json` by running the script in the file `populate.js`:
```
bash
npm run populate
- Run the development server: To start the development server, input the following command in the terminal: ``` bash npm run server
## Basic Pagination
In this section, we’ll implement basic pagination in Express.js and MongoDB without the use of any external library. We will create an API endpoint that retrieves a subset of movie records from the database based on the values of the specified query parameters. The routes have already been set up in the boilerplate code, so we’ll focus on the controller function. We will be working on the `getMoviesCustom` function, in the `controllers/movies.js` file.
We’ll use `Movie.find` cursor to fetch movies and the `skip` and `limit` methods for pagination.
* `skip`: The `skip` method controls where MongoDB starts returning records. The `skip` method has only one parameter `offset`, which is the number of records to be skipped.
* `limit`: The `limit` method defines the maximum amount of documents to be returned by MongoDB.
Two query parameters are required for basic pagination:
* `page`: This parameter specifies the page to be returned. The `page` parameter defaults to `1` if not specified. To extract the `page` query parameter input the following in the `getMoviesCustom` function:
```
javascript
const page = parseInt(req.query.page, 10) || 1;
-
limit: This parameter specifies the number of movies to return per page. Thelimitparameter defaults to 10. To extract thelimitquery parameter input the following in thegetMoviesCustomfunction: ``` javascript const limit = parseInt(req.query.limit, 10) || 10;
We need to calculate the value of the `offset` variable. As explained earlier, the `offset` variable will be passed as the parameter to the `skip` method. It is calculated based on the requested page number and the number of documents per page. To do this, add the following line in the `getMoviesCustom` function:
javascript
const offset = (page - 1) * limit;
Let’s assume we want `10` movies to be returned per page. Therefore, the `limit` variable will have a value equal to `10`. The value for the `offset` variable will be calculated as follows:
* For the first page, the `page` parameter has a value of `1`. Therefore `offset` will have a value equal to `0`. This means zero documents are skipped at the beginning.
* For the second page, the `page` parameter has a value of `2`. Therefore `offset` will have a value equal to the value of the `limit`, which is `10`. This means the first `10` documents are skipped.
* For the third page, the `page` parameter has a value of `3`. Therefore `offset` will have a value equal to the value of `2*limit`, which is `20`. This means the first `20` documents are skipped.
Similar logic applies to subsequent pages.
To retrieve a subset of movie documents based on the calculated `offset` values and `limit`, add the following line to the `getMoviesCustom` function:
javascript
const movies = await Movie.find().skip(offset).limit(limit).exec();
In our API response, we also want to return the following pieces of information:
* `totalItems`: The total number of documents in the `Movie` collection. This is done using the `countDocuments` method. In the `getMoviesCustom` function, add the following line;
```
javascript
const totalItems = await Movie.countDocuments({});
-
totalPages: The total number of pages. This is done by dividing the total of documents (totalItems), by the number of documents per page (limit), and rounding off the nearest whole number. To calculate the total number of pages, in thegetMoviesCustomfunction, add the following line: ``` javascript const totalPages = Math.ceil(totalItems / limit);
Putting it all together, the `getMoviesCustom` controller function should look like this:
javascript
async function getMoviesCustom(req, res) {
const page = parseInt(req.query.page, 10) || 1;
const limit = parseInt(req.query.limit, 10) || 10;
const offset = (page - 1) * limit;
const movies = await Movie.find().skip(offset).limit(limit).exec();
const totalItems = await Movie.countDocuments({});
const totalPages = Math.ceil(totalItems / limit);
return res.status(200).json({ totalItems, page, totalPages, movies });
}
## Using the `mongoose-paginate-v2` Library
In this section, we’ll implement pagination using the `mongoose-paginate-v2` library. An API route has already been created in the boilerplate code, so we’ll focus on the `getMoviesLibrary` controller function.
The `mongoose-paginate-v2` is a pagination library for `Mongoose`. According to the docs:
>The main usage of the plugin is you can alter the return value keys directly in the query itself so that you don't need any extra code for transformation.
To use `mongoose-paginate-v2`, we need to add the plugin to the schema and make use of the model `paginate` method. In the `model/Movie.js` file, make the following adjustments to the schema:
javascript
const mongoose = require("mongoose");
const mongoosePaginate = require("mongoose-paginate-v2");
const movieSchema = new mongoose.Schema(
...
);
movieSchema.plugin(mongoosePaginate);
module.exports = mongoose.model("Movie", movieSchema);
In the `getMoviesLibrary` controller function in `controllers/movies.js`, input the following code:
javascript
function getMoviesLibrary(req, res) {
const { page, limit } = req.query;
const options = {
page: parseInt(page, 10),
limit: parseInt(limit, 10),
const movies = Movie.paginate({}, options).then((result) => {
return res.status(200).json({
totalItems: result.totalDocs,
currentPage: result.page,
totalPages: result.totalPages,
movies: result.docs,
});
});
}
};
The Model `paginate` method returns a promise, and has three parameters:
1. `query`: The `query` parameter is an object that states the condition for filtering documents. In our case, since we are returning all records, we pass an empty object `{}`.
2. `options`: The `options` parameter is an object that contains various properties that control how the data is paginated. Some of the object properties include:
* `page`: The current page number. It automatically defaults to `1` if no value is provided.
* `limit`: The number of documents per page. It automatically defaults to `10` if no value is provided.
You can find a list of all the `options` object's properties in the [documentation](https://www.npmjs.com/package/mongoose-paginate-v2).
3. `callback(err, result)`: The parameter is an optional function that gets executed when the pagination results are returned or there is an error in the query.
The Model `paginate` method returns an object after the promise is fulfilled. The properties of the returned object provide information about the paginated result. Some of the properties of the object include:
* `docs`: An array of documents for the specified page that matches the query.
* `totalDocs`: The total number of documents in the collection that match the query.
* `page`: The current page number.
* `totalPages`: The total number of pages based on the total documents and the limit.
Visit the [documentation](https://www.npmjs.com/package/mongoose-paginate-v2) for the full list of the object properties.
The `mongoose-paginate-v2` library provides the helper class `PaginationParameters`. The `PaginationParameters` class enables passing the entire request query parameter to the `paginate` method. This abstraction eliminates the need to declare the parameters individually in the controller function. To use the helper class, replace the content of `getMoviesLibrary` with the following:
javascript
function getMoviesLibrary(req, res) {
Movie.paginate(...new PaginationParameters(req).get()).then((result) => {
return res.status(200).json({
totalItems: result.totalDocs,
currentPage: result.page,
totalPages: result.totalPages,
movies: result.docs,
});
});
}
## Testing the API
In this section, we’ll use Postman (or any other API client) to test the behavior of the endpoints. The two API endpoints created in the tutorial are:
* `/api/v1/movies/custom`: This endpoint is routed to the `getMoviesCustom` controller, which contains the logic for our basic implementation of pagination.
* `/api/v1/movies/library`: This endpoint is routed to the `getMoviesLibrary` controller, which contains the logic for our implementation of pagination using the `mongoose-paginate-v2` library.
Any request made to both endpoints with the same query parameters should give the same results.
Let’s make the following requests:
1. Make a `GET` request to `/api/v1/movies/custom `and `/api/v1/movies/library`. The `page` and `limit` parameters were not specified, so they should revert to their default values. The `page` parameter would have a value of `1`, the `limit` parameter would have a value of `10`, and a total of `3` pages. The result should look like this:
2. Make a `GET` request to `/api/v1/movies/custom?page=2&limit=4` and `/api/v1/movies/library?page=2&limit=4`. The `limit` parameter has a value of `4` and the `page` parameter has a value of `2`. The request should result in the API endpoint returning `4` documents. The response should indicate that the current page is `2` out of a total of `8` pages. (After populating the database with `MOCK_DATA.json`, there should be a total of `30` records). The result should look like this:
## Conclusion
You now have a basic understanding of how to implement server-side pagination using Express.js and MongoDB. You created two endpoints, one for basic pagination and the other for pagination using the `mongoose-paginate-v2` library.
You can explore additional query features to enhance your application, such as filtering and sorting. For further reading, check out the following resources:
* `mongoose-paginate-v2` [Documentation](https://www.npmjs.com/package/mongoose-paginate-v2).
* [Mongoose Documentation](https://mongoosejs.com/docs/guide.html).
* [MongoDB Documentation](https://www.mongodb.com/docs/).
* [Express.js Documentaion](https://expressjs.com/en/4x/api.html).
Top comments (0)