DEV Community

Jim Frenette
Jim Frenette

Posted on • Edited on • Originally published at jimfrenette.com

Hugo + Node.js Koa App Connected to MongoDB Part II

#hugo #node #koa #mongodb

In part 2 of the series, we're adding the site content to a search.json file when the Hugo static site is built and importing it into a MongoDB collection. Then in the Node.js Koa app, we create an API endpoint that returns search results from the collection in MongoDB.

This tutorial a continuation of Hugo + Node.js Koa App Connected to MongoDB and thus requires that all of the steps in it have been completed prior to starting this one. The source code for this series is on GitHub.

Environment Settings

To make things easier to change and keep track of, add an .env file to maintain project settings. For example,

.env 
### PROJECT SETTINGS

DEV_PROJECT_NAME=hkm
DEV_PROJECT_HOST=localhost
Enter fullscreen mode Exit fullscreen mode

Now update the docker-compose.yml file to use the .env file settings as follows.

  1. Replace hkm_ with ${DEV_PROJECT_NAME}_
  2. Replace localhost with ${DEV_PROJECT_HOST}

NoSQL Client

While we're updating the docker-compose.yml file, let's add a MongoDB NoSql Client Docker container to the stack so we can manage and administer MongoDB using a GUI.

In the docker-compose.yml file, add ths block below the mongo service. The mongoclient service we're adding uses the mongoclient docker image.

docker-compose.yml 
  mongoclient:
    image: mongoclient/mongoclient
    container_name: "${DEV_PROJECT_NAME}_mongoclient"
    labels:
      - 'traefik.backend=${DEV_PROJECT_NAME}_mongoclient'
    ports:
      - "3000:3000"    
    volumes:
      - mongoclientdata:/data/db
Enter fullscreen mode Exit fullscreen mode

And at the bottom of the file, add the mongoclientdata named volume under mongodata to persist any saved data changes made using the container.

docker-compose.yml 
volumes:
  mongodata:
  mongoclientdata:
Enter fullscreen mode Exit fullscreen mode

Bring up the stack using docker-compose up and open the Nosqlclient in a browser
at. For example, localhost:3000

In the menu, select Connect > Create New

Copy the connStr value from api/src/server/chinook/connect.js and paste that in the Connection URL input of the Add Connection form.

Nosqlclient Add Conection form

Switch to the chinook database and you should be able to browse the collections.

We will use this client later to import a search index json into a new database.

Search Index

These next steps will allow us to create a search index json file that can be imported into the MongoDB in order to create an api endpoint for free text search of the static website content.

Ceate a layout to output the search index json. For example, in the common /layouts folder, create a new file named list.search.json as follows:

list.search.json 
{{- $.Scratch.Add "search" slice -}}
{{- range $index, $element := (where .Site.Pages "Kind" "page") -}}
    {{- $.Scratch.Add "search" (dict "title" $element.Title "uri" $element.Permalink "tags" $element.Params.tags "section" $element.Section "content" $element.Plain "summary" $element.Summary "date" ($element.Date.Format "2006-01-02")) -}}
{{- end -}}
{{- $.Scratch.Get "search" | jsonify -}}
Enter fullscreen mode Exit fullscreen mode

Update the site configuration by adding a Search output. In order to add another output, we also need to include the pre-existing HTML and RSS output configuration settings. Update the config.toml as follows to add the Search output and format settings.

config.toml 
...

[outputs]
home = ["HTML", "RSS", "Search"]

[outputFormats.Search]
baseName = "search"
isPlainText = true
mediaType = "application/json"
notAlternative = true
Enter fullscreen mode Exit fullscreen mode

Note that an ellipsis … in the code snippets are not a part of the code and are there only to denote code that is being skipped and not applicable to the example.

Generate the site and you should now have a search.json file in the root of the public folder.

cd www
hugo
Enter fullscreen mode Exit fullscreen mode

Content

For testing, I used free public domain book excerpts which have plenty of text to populate the search index. For example, content folder

MongoDB - Import Search Index

Option A

Shell into the running MongoDB container, e.g.,

# start/restart environment as needed
docker-compose up -d

docker exec -it hkm_mongo /bin/bash
Enter fullscreen mode Exit fullscreen mode

Copy the search.json file into the mongo containers /home directory. e.g.,

docker cp www/public/search.json hkm_mongo:/home/search.json
Enter fullscreen mode Exit fullscreen mode

Import the search.json file into a database named search and a Collection named Content using the --jsonArray option. e.g.,

cd home

mongoimport --db search --collection Content --file search.json --jsonArray
Enter fullscreen mode Exit fullscreen mode

Option B (Rebuild Mongo Image)

This method requires removal of the pre-existing mongodata volume.

If the Docker image finds any files with the .sh extensions .sh inside of the /docker-entrypoint-initdb.d folder, it will execute them. Copy the search.json and a shell script to run the mongoimport command into the docker-entrypoint-initdb.d using the mongo.dockerfile. For example,

cp www/public/search.json docker/entrypoint-initdb.d/
Enter fullscreen mode Exit fullscreen mode

Create a shell script in the docker/entrypoint-initdb.d folder containing this mongoimport command:

mongoimport --db search --collection Content --file /docker-entrypoint-initdb.d/search.json --jsonArray
Enter fullscreen mode Exit fullscreen mode

Rebuild the docker image. e.g.,

# rebuild
docker-compose build
Enter fullscreen mode Exit fullscreen mode

MongoDB - Text Search

Create a text index on both the title and content fields. e.g.,

db.Content.createIndex( { title: "text", content: "text" } )
Enter fullscreen mode Exit fullscreen mode

To run the MongoDB commands, use the Nosqlclient at localhost:3000 or shell into the container, e.g., docker exec -it hkm_mongo /bin/bash and run mongo. Be sure to switch to the search database, e.g., use search

Use the $text query operator to perform a text search. e.g.,

db.Content.find( { $text: { $search: "wonderland" } } )
Enter fullscreen mode Exit fullscreen mode

Content Data Model

In the src/server/search folder, create the content data model. For example,

content.js 
const mongoose = require('mongoose');
const Schema = mongoose.Schema;

/*
 notice there is no ID. That's because Mongoose will assign
 an ID by default to all schemas

 by default, Mongoose produces a collection name by passing the model name to the utils.toCollectionName method.
 This method pluralizes the name Content to Contents.
 Set this option if you need a different name for your collection.
 e.g., `collection: 'Content'`
*/

const ContentSchema = new Schema({
    content: String,
    date: Date,
    section: String,
    summary: String,
    tags: Array
},{ 
    collection: 'Content'
});

const search = mongoose.connection.useDb('search');

module.exports = search.model('Content', ContentSchema);
Enter fullscreen mode Exit fullscreen mode

Search Route

In the src/server/routes/search folder, create the api/search route. For example,

content.js 
const Router = require('koa-router');

const router = new Router();
const BASE_URL = `/api/search`;

const Content = require('../../search/content');

function getContent(phrase) {

    // put quotes around search phrase
    phrase = JSON.stringify(phrase);

    return new Promise((resolve, reject) => {
        var query = Content.find({ $text: { $search: phrase } });
        query.exec((err, results) => {
            if (err) return handleError(err);
            resolve(results);
        });
    });
}

router.get(BASE_URL + '/content/:phrase', async (ctx) => {
    try {
        ctx.body = await getContent(ctx.params.phrase);
    } catch (err) {
        console.log(err)
    }
})

module.exports = router;
Enter fullscreen mode Exit fullscreen mode

App Entrypoint

Update the Koa application entrypoint, src/server/index.js to include and use the new content text search route.

index.js 
const Koa = require('koa');
const mongoose = require('mongoose');

...

const contentRoutes = require('./routes/search/content');

/**
 * Koa app */
const app = new Koa();
const PORT = process.env.PORT || 1337;
const server = app.listen(PORT, () => {
    console.log(`Server listening on port: ${PORT}`);
});

/**
 * MongoDB connection */
const connStr =  'mongodb://mongo:27017/default';
mongoose.connect(connStr);
const db = mongoose.connection;
db.on('error', console.error.bind(console, 'connection error:'));
db.once('open', () => {
    console.log('connected');
});

...

app.use(contentRoutes.routes());

module.exports = server;
Enter fullscreen mode Exit fullscreen mode

API Test

In the project root, run docker-compose up -d which starts the containers in the background and leaves them running. The -d is for detached mode.

To monitor docker output in the terminal, run docker-compose up without the disconnected (-d) option, e.g., docker-compose up.

To test the content search API endpoint, load localhost/api/search/content/white rabbit in a browser to see the json response. In this example, we're searching for "white rabbit".

content search API json response

All of the source code for this tutorial is available on GitHub.

Source Code

Originally published at jimfrenette.com/2019/08/hugo-plus-nodejs-koa-app-mongodb-connection-part-2

Top comments (0)

Read next

philipwalsh profile image

React Best Practices with Examples

Philip Walsh -

atimin profile image

Keeping MQTT Data History with Node.js

Alexey Timin -

thinkthroo profile image

Node.js util module usage in Changesets

thinkThroo -

juniourrau profile image

🦸 Supercharge your debugging skills with the console.trace

Ravin Rau -