John is a member of the NestJS core team
Ever wanted to integrate your favorite library into NestJS? For example, while Nest has broad built-in support for database integration, what if you want to use your own favorite library and no Nest package exists? Well, why not build your own?
This might seem like a daunting task at first. But if you've been following my blog posts, you saw a design pattern for NestJS dynamic modules in my last post that points you in the right direction. OK, you say, but it still seems like a lot of work to integrate a library.
I have good news! Using the awesome power of the Nest CLI, you can generate a complete, custom dynamic module template, following NestJS best practices, with a single command! You can then literally have your library integrated in about 5 minutes! This technique works for many resource-based libraries that export what I'll call an API object, which naturally work well with Nest's built-in singleton provider model for sharing a global resource. Follow along with such an adventure as we build a quick integration to Knex.js below.
And stay tuned for more on the wonders of the vastly under-appreciated NestJS CLI. Built on top of the Angular CLI, the potential for this tool to revolutionize the way you use Nest is limitless. I have lots more planned for this topic in the near future!
Intro
In my How to build completely dynamic NestJS modules article last week, I covered a design pattern that is used throughout standard Nest packages like @nestjs/passport
, @nestjs/jwt
, and @nestjs/typeorm
. The point of the article was two-fold:
- Provide a roadmap to a basic pattern used within common NestJS packages to help you be a better consumer of those packages.
- Encourage Devs to think about how to make use of this pattern in their own code to enable more easily composable modules.
Some might consider the pattern a bit complex, and I wouldn't necessarily argue. However, gaining a good understanding of it is useful, and helps in the quest for mastery of some core concepts of Nest, especially around leveraging the module system and dependency injection. Nevertheless, there's a fair amount of boilerplate code in the pattern. Wouldn't it be great to simplify the process?
Buckle up! NestJS CLI custom schematics to the rescue! 🚀
What We'll Build
In this tutorial, we'll build a module that exports a direct API to the full Knex.js library. This is a powerful DB integration library used widely across the Node.js ecosystem. Here's what we'll do. The following represent the exact same steps you can use to integrate any other basic callable API (for example, ioredis, Cassandra, Neo4J, Elasticsearch, LevelDb to name just a few).
- Use the dynpkg custom schematic to generate a customized package (the schematic automates the dynamic module pattern I've been discussing above).
- Fill in a few quick details to customize the package for Knex.js.
- Use the auto-generated test client to demonstrate that it works.
Thanks to the power of the Nest CLI, coupled with a custom schematic, we can complete these steps in about 5 minutes! As a bonus, we'll cover a few more advanced features at the end of this article, including how to publish the generated package on npm with a single click.
If you want to see a completed version of the steps presented in the remainder of this article (a fully useable NestJS/Knex.js module), you can find a github repo with the completed code here.
Nest CLI and Schematics
You probably already use the Nest CLI on a regular basis. nest new myProject
and nest generate controller user
are common examples. They use the @nestjs/schematics
package that ships with the CLI out-of-the-box. What's very cool is that you can easily use other schematics to do highly customized work. A feature common to all schematics is that they understand your project, and have a smart way of scaffolding new architectural components and wiring them into your project. Think of an individual schematic as a blueprint, and of the CLI machinery as a set of standards for how each blueprint works. Because of this, new schematics "just work", and inherit all of the features provided by the CLI and its standard capabilities.
You can write your own schematics, and I plan to show you how in some upcoming blog posts. For now, I've built one that you can use to do the library integration project we've undertaken today. Let's get started!
To use any external schematics, you of course need to install them. One important note: you must install a schematics collection as a global package, due to the way the CLI works.
Installing a Custom Schematics Collection
Schematics are packaged as collections and can be bundled up as npm packages, so installing them is simple. You can do so now with:
npm install @nestjsplus/dyn-schematics -g
Using a Custom Schematics Collection
Using schematics from a custom collection is straightforward. The standard Nest CLI command structure looks like this:
nest commandOrAlias [-c schematicCollection] requiredArg [options]
-
commandOrAlias
is:new
orgenerate
(alias:g
) oradd
-
schematicCollection
is optional, and defaults to the built-in NestJS schematics; you can optionally specify a globally installed npm package -
requiredArg
is the architectural element being generated/added, e.g.,controller
,module
, or soon, in our casedynpkg
-
options
are global to all schematics; they can be--dry-run
,--no-spec
, or--flat
Once it's installed, you use the @nestjsplus/dyn-schematics package like this:
- Make sure you're in the folder you want to have as the parent of the project. With this schematic, we're creating a new complete nest package, meaning it's a new standalone project. So it will create a folder using the
name
you provide, and put all of the component parts inside that folder. - Run the schematic with the CLI
nest g -c @nestjsplus/dyn-schematics dynpkg nest-knex
This runs using the custom schematics collection from @nestjsplus/dyn-schematics
and specifying the dynpkg
schematic to execute (the schematic identifies the thing to generate - in this case, a dynamic module package identified as dynpkg
), and giving it a name
of nest-knex
. If you want to see what this would do without adding files to your filesystem, just add --dry-run
at the end of the command.
This should prompt you with the question Generate a testing client?
. Answer yes to this to make testing easier. You'll see the following output, and you'll then be able to open this project in your IDE from the nest-knex
folder.
► nest g -c @nestjsplus/dyn-schematics dynpkg nest-knex
? Generate a testing client? Yes
CREATE /nest-knex/.gitignore (375 bytes)
CREATE /nest-knex/.prettierrc (51 bytes)
CREATE /nest-knex/README.md (2073 bytes)
CREATE /nest-knex/nest-cli.json (84 bytes)
CREATE /nest-knex/package.json (1596 bytes)
CREATE /nest-knex/tsconfig.build.json (97 bytes)
CREATE /nest-knex/tsconfig.json (430 bytes)
CREATE /nest-knex/tslint.json (426 bytes)
CREATE /nest-knex/src/constants.ts (54 bytes)
CREATE /nest-knex/src/index.ts (103 bytes)
CREATE /nest-knex/src/main.ts (519 bytes)
CREATE /nest-knex/src/nest-knex.module.ts (1966 bytes)
CREATE /nest-knex/src/nest-knex.providers.ts (262 bytes)
CREATE /nest-knex/src/nest-knex.service.ts (1111 bytes)
CREATE /nest-knex/src/interfaces/index.ts (162 bytes)
CREATE /nest-knex/src/interfaces/nest-knex-module-async-options.interface.ts (532 bytes)
CREATE /nest-knex/src/interfaces/nest-knex-options-factory.interface.ts (194 bytes)
CREATE /nest-knex/src/interfaces/nest-knex-options.interface.ts (409 bytes)
CREATE /nest-knex/src/nest-knex-client/nest-knex-client.controller.ts (732 bytes)
CREATE /nest-knex/src/nest-knex-client/nest-knex-client.module.ts (728 bytes)
At this point, you can install the generated code:
cd nest-knex
npm install
Now start up the app with:
npm run start:dev
If you now browse to http://localhost:3000 you should get:
Hello from NestKnexModule!
Great! At this point, we've scaffolded the framework of our dynamic module, and we're ready to begin the Knex.js integration.
Integrating with Knex.js
You now have a project scaffolded to do the library integration. We have to make just a few edits to perform the actual integration. Let's talk them through:
- We need to install
knex
as a dependency. Note, in order to run this project, you'll need access to a live SQL database. Knex.js doesn't bundle any database libraries, so you'll need to install the appropriate one (you can read more here). In this tutorial, I'll use PostgreSql, which I have available on localhost. If you do not have a local DB available, you can consider using a docker setup. - We need a way to provide the configuration options to the Knex.js API.
- We need to call the Knex.js API in the way it wants to be initialized, returning a handle to a
knex
object which we can use to access Knex.js features. - To test things out, we can use the client controller that was auto-generated by the schematic.
- Once it works, we can use this package directly. Or we can publish the package to a registry (e.g., an internal package registry or publicly to npmjs.com)
Install Dependencies
The knex
package is required. You will also need a database API library to run the module. Below, I use pg
for PostgreSql, but you can choose whatever you want (from the list here).
cd nest-knex
npm install knex pg
Knex.js Options
As discussed in the dynamic modules article, we provide options to the service using an async options provider. The code to do all this has been scaffolded for you, but to pass the appropriate options in a native TypeScript way, we need to modify the NestKnexOptions
interface. This is contained in the file src/interfaces/nest-knex-options.interface.ts
.
We want to describe the available Knex.js options that will be passed to the library API. The easiest way to do this is to use the types already provided by Knex.js. Since this interface is exported by the knex
package, we can simply import it, and our job is nearly done! We'll simply alias it to make it visible to our generated package. Open up src/interfaces/nest-knex-options.interface.ts
and edit it to look like this:
// src/interfaces/nest-knex-options.interface.ts
import { Config } from 'knex';
export interface NestKnexOptions extends Config {}
Knex Connections
The job of our module is pretty simple: connect to the API and return a re-usable knex
object for interacting with the database. In the case of PostgreSql, knex
(on top of pg
) returns a connection pool, which means we can connect once and return a singleton knex
object that will automatically balance multiple queries across the connection pool. This works seamlessly with NestJS and its notion of singleton providers!
Note: I'm less familiar with Knex than other DB libraries, and also less familiar with other DBs like MySQL, so if you are using this module for such databases, be sure you understand the right pattern for sharing a
knex
connection object within your application. This topic is not a NestJS issue -- it's all about the architecture of the Knex.js library and the database client library.
Now we'll implement a simple pattern to return our knex
object. Open up src/nest-knex.service.ts
. Have a quick look to see what the generated code looks like, then replace that code with the following:
// src/nest-knex.service.ts
import { Injectable, Inject, Logger } from '@nestjs/common';
import { NEST_KNEX_OPTIONS } from './constants';
import { NestKnexOptions } from './interfaces';
const Knex = require('knex');
interface INestKnexService {
getKnex();
}
@Injectable()
export class NestKnexService implements INestKnexService {
private readonly logger: Logger;
private _knexConnection: any;
constructor(@Inject(NEST_KNEX_OPTIONS) private _NestKnexOptions: NestKnexOptions) {
this.logger = new Logger('NestKnexService');
this.logger.log(`Options: ${JSON.stringify(this._NestKnexOptions)}`);
}
getKnex() {
if (!this._knexConnection) {
this._knexConnection = new Knex(this._NestKnexOptions);
}
return this._knexConnection;
}
}
Almost all of this is unchanged from the generated boilerplate. We simply added a property to store our connection object (_knexConnection
), and a getKnex()
method to instantiate the object the first time it's requested, and then cache it for future use.
Test the Module
As mentioned, you'll need a local database instance available to test the module. If you don't have one handy, consider using docker to do so. It's easy!
The @nestjsplus/dyn-schematics
dynpkg
schematic built a small test client module for us (assuming you answered yes
to the prompt). We can use this to quickly test our nest-knex
module.
First, open src/nest-knex-client/nest-knex-client.module.ts
and add the needed Knex.js options in the register()
method. Adjust yours appropriately:
// src/nest-knex-client/nest-knex-client.module.ts
...
@Module({
controllers: [NestKnexClientController],
imports: [
NestKnexModule.register({
client: 'pg',
connection: {
host: 'localhost',
user: 'john',
password: 'mypassword',
database: 'nest',
port: 5432,
},
}),
],
})
...
Now, open src/nest-knex-client/nest-knex-client.controller.ts
and plug in some queries. Of course this is not really a great design pattern (invoking database services directly from your controller), but is really just a quick test that the Knex.js integration works. In reality, you'll want to delegate any such access to true NestJS services as per NestJS best practices.
Here's an idea of what you can try. This test relies on the following database table being available (syntax below works with PostgreSql, but may require slight tweaks for other databases):
CREATE TABLE cats
(
id serial NOT NULL,
name text,
age integer,
breed text
);
ALTER TABLE cats
ADD CONSTRAINT cats_pkey
PRIMARY KEY (id);
And here's a sample of what you could try in your test controller. Note that you have the full power of Knex.js available via the knex
object. See here for lots more interesting samples of things you can do with Knex. Since you have a handle to the knex
object, all of those Knex Query Builder methods should just work! In this admittedly silly example, we are both creating and querying some cats in our index route, using the knex
object to access the database.
// src/nest-knex-client/nest-knex-client.controller.ts
import { Controller, Get } from '@nestjs/common';
import { NestKnexService } from '../nest-knex.service';
@Controller()
export class NestKnexClientController {
constructor(private readonly nestKnexService: NestKnexService) {}
@Get()
async index() {
const knex = this.nestKnexService.getKnex();
const newcat = await knex('cats').insert({
name: 'Fred',
age: 5,
breed: 'tom cat',
});
const cats = await knex.select('*').from('cats');
return cats;
}
}
Now, fire up the app with npm run start:dev
, and browse to http://localhost:3000
, and you should see the results of your query!
If you want to test the module with a separate app -- really demonstrating the power of the reusable library module you just built -- read on to see how to upload it as an npm package in one step. And for convenience, you can download a full client app that imports the module here and use it to test your newly minted Knex.js module. In fact, knex-cats is a fully working example, so if you've been reading along without coding, that's the easiest way to see the finished product.
Bonus Section
Hopefully I kept my promise of showing you how quickly you can generate a dynamic NestJS module that wraps an external resource library in just a few minutes! We're done -- you have a full-fledged Knex.js API at your disposal now! The queries we did in the previous section are just the tip of the iceberg, as Knex.js has a boat load of cool features.
Let's cover a couple more topics to round out the discussion.
A Better API
As mentioned, it's not good practice to access the knex
object directly from our controller. We can make things a little bit better by creating a service to house our database logic. We can also make our Knex.js module a little easier to use by adding a higher level API. Let's do that. We're going to add a provider that let's us directly inject the knex
object into any service. You'll see how this cleans the API up in a moment.
First, create a file called nest-knex-connection.provider.ts
inside the src
folder. Add the following code:
// src/nest-knex-connection.provider.ts
import { KNEX_CONNECTION } from './constants';
import { NestKnexService } from './nest-knex.service';
export const connectionFactory = {
provide: KNEX_CONNECTION,
useFactory: async nestKnexService => {
return nestKnexService.getKnex();
},
inject: [NestKnexService],
};
To follow best practices for providers, we are using the KNEX_CONNECTION
constant as our provider injection token. Be sure to add this line to src/constants.ts
:
export const KNEX_CONNECTION = 'KNEX_CONNECTION';
See what we're doing here? We're binding an injection token (KNEX_CONNECTION
) to a factory so that we can use our knex
API object directly, avoiding the need to instantiate the service. For example, once we complete one more step (below), we'll be able to access the knex
object in our controller as follows (notice the slightly simplified technique between old and new in the sample code):
// src/nest-knex-client/nest-knex-client.controller.ts
import { Controller, Inject, Get } from '@nestjs/common';
import { KNEX_CONNECTION } from '../constants';
@Controller()
export class NestKnexClientController {
// new
constructor(@Inject(KNEX_CONNECTION) private readonly knex) {}
// old
// constructor(private readonly nestKnexService: NestKnexService) {}
@Get()
async index() {
// following line no longer needed
// const knex = this.nestKnexService.getKnex();
const newcat = await this.knex('cats').insert({
name: 'Fred',
age: 5,
breed: 'tom cat',
});
const cats = await this.knex.select('*').from('cats');
return cats;
}
}
To wire this new provider into our module, open up src/nest-knex.module.ts
and make the following changes:
- Import the
connectionFactory
- Add the
connectionFactory
to the@Module
metadataproviders
andexports
properties. This section of the file will now look like this:
import { connectionFactory } from './nest-knex-connection.provider';
@Global()
@Module({
providers: [NestKnexService, connectionFactory],
exports: [NestKnexService, connectionFactory],
})
The complete implementation of our best practices for database access would also have us create a service that accesses the KNEX_CONNECTION
provider, rather than doing that in a controller, but I'll leave that as an exercise for the reader. 😉
Dynamic Registration (with a Config Factory)
In the dynamic modules article, we talked about using asynchronous options providers. By that, we meant that rather than use a statically declared object containing our connection params like this:
NestKnexModule.register({
client: 'pg',
connection: {
host: 'localhost',
user: 'john',
password: 'mypassword',
database: 'nest',
port: 5432,
},
}),
We'd instead like to supply our connection options in a dynamic fashion. Good news! This is already built-in to the generated package. Let's test it out. For simplicity, we'll just use an in-place factory, but you can use the full power of class-based, factory based, and existing providers. To test it, we'll update the registration of the NestKnexModule
in src/nest-knex-client/nest-knex-client.module.ts
to look like this:
// src/nest-knex-client/nest-knex-client/module.ts
NestKnexModule.registerAsync({
useFactory: () => {
return {
debug: false,
client: 'pg',
connection: {
host: 'localhost',
user: 'john',
password: 'mypassword',
database: 'nest',
port: 5432,
},
};
},
}),
It's not a terribly exciting example, but you get the idea. You can take advantage of any of the asynchronous provider methods to supply the configuration data to the module. If you check out the knex-cats sample, you'll see some more robust use of a config module to configure our Knex module dynamically.
Publish the Package
If you read my earlier Publishing NestJS Packages with npm article, you may have noticed that the structure of this package (e.g., the package.json
file, the tsconfig.json
file, the presence of the index.ts
file in the root folder) follows that pattern exactly. As a result, publishing this package is as simple as this (assuming you have an npmjs.com
account, that is):
npm publish
Seriously, that's it!
Of course, you can always use this ready-made @nestjsplus/knex package, built exactly as describe in this article.
Not Just External APIs!
The @nestjsplus/dyn-schematics package not only supports generating a standalone package, as described so far, but also generating a regular dynamic module inside an existing project. I won't cover that in detail here, but the use-case is simple. Let's say you're building a ConfigModule for use within a project. Simply scaffold it with the dynmod
schematic (instead of dynpkg
), and it will generate a new dynamic module into your existing project, fully ready for you to implement the details of your register()
and registerAsync()
static methods.
This works exactly like the normal CLI commands for adding elements like controllers, services, and modules. In fact, its behavior is very parallel to nest generate module
. It builds the dynamic module and all its parts (which look quite similar to what we've seen in this article, minus the top level Nest application components) in a folder inside your project, and wires them up to the rest of the project just as the normal module
schematic does. Read more about it at the github page for @nestjsplus/dyn-schematics. This use-case (adding a dynamic module to an existing project) is covered here. You can easily give it a quick dry run inside an existing Nest project with:
nest g -c @nestjsplus/dyn-schematics dynmod myNewModule --dry-run
Conclusion
If you've been following this series, you've previously learned a powerful architectural pattern for building modular NestJS services and modules, and composing them into applications. Now, with the power of schematics, you can automatically generate the boilerplate code to implement this powerful pattern. You can use this schematic to customize your own internal modules, or to easily integrate external APIs. In future articles, I'll show you how to build your own schematics to extend the power of NestJS and the NestJS CLI even further!
Feel free to ask questions, make comments or suggestions, or just say hello in the comments below. And join us at Discord for more happy discussions about NestJS. I post there as Y Prospect.
Top comments (19)
Great article. I'm using Typescript and it was giving me this error in the
getKnex()
function:did a bit of quick research and the fix is this:
Keep up the great work!
dev.to/umasankarswain/how-to-updat...
Another great article @johnbiundo 👏 This series is really interesting. Thanks for you work 👍
I have a few questions that might sound dumb but I'm going to ask them anyway:
Global
by default. Is that the way to go? Shouldn't we avoid global modules? Let's say I need a specific connection to a Postgres instance in module A and another connection to another instance in module B. If theNestKnexModule
is global, won't that be an issue?register
andregisterAsync
methods: I thought that global modules were supposed to exposeforRoot
(and its async variant) and that module-scoped modules were supposed to exposeregister
(and its async variant). Have I been wrong this whole time? Or is it just a matter of personal preference?Again, thanks for your work! I wish those schematics were released sooner as they would have saved me some time :)
Can't wait for your next post.
@fwoellfel, thanks! Glad it has been helpful!
Not dumb at all! I should probably discuss the use of
@Global()
in the article, so thanks for bringing it up. The main reason I use it is, to be honest, in a "utility" type module I find it convenient to register it once (usually in the root module), and then have it globally available. However you bring up an excellent point, and@Global()
definitely has some side effects (such as breaking module encapsulation) and is usually to be avoided. I want to give this some more thought, and I have an idea or two on how to handle your Module A/Module B question, but need to test a few things out. I'll report back!And your second question is good as well! To be honest, I hadn't quite sussed out that pattern of
register
vs.forRoot
, but now that you mention it, I'll have to go back and look more closely.Thanks so much for the thoughtful feedback. It really helps to have sharp eyes from people who have worked on dynamic modules. I hope to continue to improve these articles as a useful resource!
dev.to/umasankarswain/how-to-updat...
I am getting this error from the
/.constants
. Why is this throwing an error for a constant?Hi. For the future readers;
Error complains about NestKnexService can not resolve injected dependency. In this case the dependancy is NEST_KNEX_OPTIONS which is a custom provider. To make the dependency injection work, the provider needs to be registered in the knex module. NEST_KNEX_OPTIONS provider is not included in this article. It is included in github link author provided.
Hi Mike,
Can you provide some more context for this error? Do you have a repository that you can share?
Best,
John
hi, im new to nextjs and knex and im trying to build a custom api with both of these apps.
However, while following your guide, after running npm install im getting the following error
npm ERR! nest-knex@1.0.0 build:
tsc
npm ERR! Exit status 2
which are related to jest-diff module.
Can u maybe point me out any directions on how to fix this and finish the current integration of knex with nestjs?
ty
apparently this error seems to be a conflict between @types/jest and typescript versions, and can be solved by updating both with jest v26.0.4 and typescript above v3.8 (v3.9.6 currently).
sry for the bother
Good afternoon, I am using your package github.com/nestjsplus/knex
and I wanted to wrap the user.service in a repository and I got errors
Potential solutions:
Have you tried newer version of knex ^1.0.3 ?
There are some breaking changes for me line there is no exported interface "Config".
is it possible to get knex debug data into knexmodule?
Can you update this article? It seems out of date.
What is in constants.ts ?
And I get error 'knex_1.Knex is not a constructor'
Maybe a working git repo would be nice. Something is missing.
another awesome post from John :)
follow the tutorial and works in current date only fails in bellow fix
with
and one must update this dependencies
thanks John
Can't seem to get past "nest g -c @nestjsplus/dyn-schematics dynpkg nest-knex".
Having the same issue, did you manage to get the package to work?
I think I did, I will have to read through my code and find out. I'll get back to you
dev.to/umasankarswain/how-to-updat...