Today, you will learn how to build a search engine using Algolia in Next.js. This is part 1 where we will handle the backend part. In part 2, we will handle the frontend part.
Table of Contents
- What is Algolia?
- How Algolia Works?
- Syncing Data with Algolia
- Setup Database
- Setup Algolia
- Listen for Changes
- Add a record to Algolia
- Update a record in Algolia
- Delete a record in Algolia
- Add existing data to Algolia
What is Algolia?
Algolia is a search engine API that allows you to build a fast and relevant search experience for your users. It provides a powerful search engine that can be easily integrated into your application. You might have seen it on documentation websites like React, Tailwind, Redux, etc.
I recommend checking the video tutorial on my youtube channel to better understand.
How Algolia Works?
A regular query to a database can be slow and inefficient, especially when dealing with large datasets.
Algolia works by indexing your data. Indexing means storing some additional data about your data in the Database. This additional data is optimized for search queries, making it much faster and more efficient than a regular query.
To do indexing, Algolia needs your data from the Database. So, you need to copy the relevant data from your Database to Algolia. Never copy secret data like passwords, credit card numbers, etc.
Records: A record is a single piece of data that you want to search for, such as a product, a user, etc. Think of it as a document in MongoDB or a row in an SQL database.
Index: It is a collection of records. Think of it like a table in an SQL database or a collection in MongoDB. An Algolia project can have multiple indexes, such as users_index, products_index, etc.
Syncing Data with Algolia
Since you are copying data from your Database to Algolia, you need to keep the data in sync. This means that whenever you add, update, or delete data in your Database, you need to update the corresponding data in Algolia. Otherwise you will have outdated/wrong search results. You can do this by:
Cron Job: You can run a cron job that periodically syncs the data between your Database and Algolia.
I have a detail video on Cron Jobs in case you are interested.
Manual Sync: You can manually sync the data whenever you make changes to your Database. Let's say you have a function updateData
the modifies the data in your Database. YOu can call another function that will apply the same changes to Algolia. You must do the same thing for every function that modifies the data.
Event-Driven Sync: You can listen for changes in your Database. Every time you create, update, or delete, a function will be triggered. There, you apply the same changes to Algolia. This will depend on the DB. You can use ChangeStream
in MongoDB. We are going to use this approach. But I haven't found a way to do this on Nextjs, so I will use a separate node server for that.
Setup Database
You can check the following guide to setup mongodb.
Setup Algolia
- Create a new account on Algolia
- A new application will be created asking you to add data. You can skip that for now.
- Get keys:
- Go to API keys page
- Copy application_id and admin_key. Admin key is secret so only use it on server and never expose.
Listen for Changes
Let's create a simple node server with express:
const { searchClient } = require('@algolia/client-search')
const mongoose = require('mongoose')
const express = require('express')
require('dotenv').config()
const app = express()
const client = searchClient(
process.env.ALGOLIA_APP_ID,
process.env.ALGOLIA_ADMIN_KEY,
)
mongoose
.connect(process.env.MONGO_URI, {})
.catch(console.error)
.then(() => console.log('Connected to MongoDB'))
// A dummy model
const User = mongoose.model('User', new mongoose.Schema({}))
const changeStream = User.watch()
const USERS_INDEX = 'users_index_yt'
changeStream.on('change', async data => {
console.log(data)
const {
operationType,
fullDocument,
documentKey: { _id },
} = data
const documentId = _id.toString()
switch (operationType) {
case 'insert':
break
case 'update':
break
case 'delete':
break
default:
break
}
})
app.listen(8000, () => {
console.log('Server running on port 8000')
})
Explanation:
- Create a
searchClient
with Algolia keys. - Connect to MongoDB and create a dummy
User
model. - Create a changeStream on the
User
model. This will listen for changes in theUser
collection. - Listen for changes using the
on
method and pass thechange
string. Then, pass a callback function that will be called whenever there is a change in theUser
collection. -
data
will contain the change information. We are interested inoperationType
,fullDocument
, anddocumentKey
.
Add a record to Algolia
const addToIndex = async data => {
try {
const { _id, name, email } = data
const userObject = {
objectID: _id.toString(),
name,
email,
}
await client.saveObject({
indexName: USERS_INDEX,
body: userObject,
})
} catch (error) {
console.log(error)
}
}
switch (operationType) {
case 'insert':
await addToIndex(fullDocument)
break
}
Explanation:
- We are extracting the
_id
,name
, andemail
from thefullDocument
. - We are creating a
userObject
with theobjectID
,name
, andemail
. - Then, we call the
saveObject
method on theclient
object. We are passing theindexName
and thebody
object.saveObject
method will add a new record to the index if there is no record with the sameobjectID
. If there is a record with the sameobjectID
, it will replace the record. Also if theindexName
doesn't exist, it will create a new index.
Update a record in Algolia
const updateIndex = async (id, data) => {
try {
const { password: _, ...updatedFields } = data
await client.partialUpdateObject({
indexName: USERS_INDEX,
objectID: id,
attributesToUpdate: updatedFields,
})
} catch (error) {
console.log(error)
}
}
switch (operationType) {
case 'update':
const {
updateDescription: { updatedFields },
} = data
await updateIndex(documentId, updatedFields)
break
}
Explanation:
- We are extracting the
updatedFields
from thedata
. Only the updated fields will be present. - Then call the
partialUpdateObject
method on theclient
object. Pass theindexName
,objectID
, andattributesToUpdate
.
Delete a record in Algolia
const deleteFromIndex = async id => {
try {
await client.deleteObject({
indexName: USERS_INDEX,
objectID: id,
})
} catch (error) {
console.log(error)
}
}
switch (operationType) {
case 'delete':
await deleteFromIndex(documentId)
break
}
Explanation:
- Call the
deleteObject
method and use the document id asobjectID
.
Add existing data to Algolia
You can add a simple script to add existing data to Algolia:
const processRecords = async () => {
await mongoose.connect(MONGO_URI)
console.log('Connected to MongoDB')
let users = await User.find({}, 'name email')
const usersObject = users.map(user => {
const { _id, ...restUser } = user.toObject()
return {
objectID: _id.toString(),
...restUser,
}
})
return await client.saveObjects({
indexName: 'users_index_yt',
objects: usersObject,
})
}
Explanation:
- Get all the data from DB
- This time, use the
saveObjects
method to add multiple records at once. It is the same assaveObject,
but you can pass an array of objects. - If adding a record fails, no other records will be added. So, you won't have partial data in Algolia.
That's it. In part 2, we will learn how to add a search bar in Next.js and use Algolia to search the data.
Thank you for reading. If you have any questions, please leave a comment below.
Make sure to check out my YouTube channel and subscribe for more tutorials! 🎥✨
Top comments (0)