DEV Community

Cover image for Fast and Efficient Pagination in Golang and MongoDB
Neeraj Kumar
Neeraj Kumar

Posted on • Edited on

1

Fast and Efficient Pagination in Golang and MongoDB

MongoDB is a document based data store and hence pagination is one of the most common use case of it. So when do you paginate the response? The answer is pretty neat; you paginate whenever you want to process result in chunks. Some common scenarios are

  • Batch processing

  • Showing huge set of results on user interfac

Paginating on client and server side are both really very expensive and should not be considered. Hence pagination is generally handled at database level and databases are optimized for such needs to

2 approaches through which you can easily paginate your MongoDB responses. Sample Document

 {
        "_id" : ObjectId("6936d17263623919cd5145db"),
        "name" : "Neeraj Kumar",
        "age" : 25
    }
Enter fullscreen mode Exit fullscreen mode

Approach 1: Using cursor.skip and cursor.limit
MongoDB cursor has two methods that makes paging easy; they are

  • cursor.skip()
  • cursor.limit()

skip(n) will skip n documents from the cursor while limit(n) will cap the number of documents to be returned from the cursor. Thus combination of two naturally paginates the response.
In Mongo Shell your pagination code looks something like

 // Page 1
    db.students.find().limit(10)

    // Page 2
    db.students.find().skip(10).limit(10)

    // Page 3
    db.students.find().skip(10).limit(10)
Enter fullscreen mode Exit fullscreen mode

implement pagination:

func GetPagination(limit, page int) error {
  ctx, cancel := context.WithTimeout(context.Background(), 12*time.Second)
  defer cancel()
   coll := o.db.Database(mongoDatabaseName).Collection(offerCollectionName)

   l := int64(limit)
     skip := int64(page * limit - limit)
     fOpt := options.FindOptions{Limit: &l, Skip: &skip}

     curr, err := coll.Find(ctx, bson.D{{}}, &fOpt)
   if err != nil {
      return result, err
   }

   for curr.Next(ctx) {
      var el Offer
      if err := curr.Decode(&el); err != nil {
         log.Println(err)
      }

      result = append(result, el)
   }

   return result, nil
}


Enter fullscreen mode Exit fullscreen mode

Approach 2: Using _id and limit

This approach will make effective use of default index on _id and nature of ObjectId. I bet you didn’t know that a Mongodb ObjectId is a 12 byte structure containing

Using this property of ObjectId and also taking into consideration the fact that _id is always indexed, we can devise following approach for pagination:

  • Fetch a page of documents from database
  • Get the document id of the last document of the page
  • Retrieve documents greater than that id
    // Page 1
    db.students.find().limit(10)

    // Page 2
    last_id = ...  # logic to get last_id
    db.students.find({'_id': {'$gt': last_id}}).limit(10)

    // Page 3
    last_id = ... # logic to get last_id
    db.students.find({'_id': {'$gt': last_id}}).limit(10)
Enter fullscreen mode Exit fullscreen mode
func GetPagination(limit, page int)  error {
    ctx, cancel := context.WithTimeout(context.Background(), 12*time.Second)
  defer cancel()

   coll := o.db.Database(mongoDatabaseName).Collection(offerCollectionName)
   ctx, _ := context.WithTimeout(context.Background(), contextTimeout)

   curr, err := coll.Find(ctx, bson.D{{}}, newMongoPaginate(limit,page).getPaginatedOpts())
   if err != nil {
      return result, err
   }

   for curr.Next(ctx) {
      var el Offer
      if err := curr.Decode(&el); err != nil {
         log.Println(err)
      }

      result = append(result, el)
   }

   return result, nil
}

Enter fullscreen mode Exit fullscreen mode

Image of Wix Studio

2025: Your year to build apps that sell

Dive into hands-on resources and actionable strategies designed to help you build and sell apps on the Wix App Market.

Get started

Top comments (0)

Imagine monitoring actually built for developers

Billboard image

Join Vercel, CrowdStrike, and thousands of other teams that trust Checkly to streamline monitor creation and configuration with Monitoring as Code.

Start Monitoring

👋 Kindness is contagious

Engage with a sea of insights in this enlightening article, highly esteemed within the encouraging DEV Community. Programmers of every skill level are invited to participate and enrich our shared knowledge.

A simple "thank you" can uplift someone's spirits. Express your appreciation in the comments section!

On DEV, sharing knowledge smooths our journey and strengthens our community bonds. Found this useful? A brief thank you to the author can mean a lot.

Okay