Table of Content | Read Next: Part 5: Plan for efficient data storage and search performance in Elasticsearch
Resources
Would you rather watch a video to learn this content? Click on the link below!
The following are the project directories shared in the blog. Click on the link below to access it!
- Code for connecting to Elastic Cloud with basic authentication
- Code for connecting to Elastic Cloud with an API Key
Connect Node.js Server to Elastic Cloud
In part 3, we signed up for a free trial of Elastic Cloud and created a deployment.
This deployment consists of Elasticsearch cluster, Kibana, Fleet, three Elastic Solutions(Enterprise Search, Observability, Security) and more!
Connecting to this deployment will give us access to everything mentioned above, including our Elasticsearch cluster.
We can connect our server to Elastic Cloud deployment via two different authentication mechanisms:
- basic authentication
- API key
API key is the best practice for production because it gives users flexibility in creating different API keys for different systems or applications.
In order to generate an API key, we need to log into Elastic Cloud as a user first(basic authentication).
Let's get started!
Step 1: Pull up the earthquake_app
directory with your code editor
In part 2, we created a project directory called the earthquake_app
and we built a Node.js server within it.
Next, we will create a file where we will store Elastic Cloud access credentials.
Step 2: Set up the config
directory
In part 2, we have installed a dependency called config
.
The config
dependency allows you to store node configuration in an external configuration file. It can be customized based on different environments(local/dev/qa/prod/etc).
In our case, we will be using config
to store access credentials such as Elastic cloud ID, username, password, and API key.
Within the earthquake_app
directory, create a config
directory. Within the config
directory, create a default.json
file as shown below.
Step 3: Enter the Elastic Cloud access credentials in the default.json
file
Copy and paste the following code into the default.json
file.
//in config/default.json
{
"elastic": {
"cloudID": "DEPLOYMENT_NAME:CLOUD_ID_DETAILS",
"username": "elastic",
"password": "LONGPASSWORD"
}
}
Your default.json
file should look like the following screenshot.
We will replace "DEPLOYMENT_NAME:CLOUD_ID_DETAILS"(line 3) and "LONGPASSWORD"(line 5) with our Elastic Cloud deployment access credentials.
In part 3, we have downloaded and saved the access credentials.
Find and open the saved credentials.
You should see something like this.
Copy your password and paste it into line 5 "password" as shown below.
Next, let's find our "DEPLOYMENT NAME:CLOUD_ID_DETAILS".
In part 3, we learned how to get to the Kibana home page.
From the Kibana home page, click on the menu
icon(red box).
It will display a drop down menu as shown below.
From the drop down menu, click on the Manage this deployment
option(pink box).
You will see the following on your screen.
Look at the region highlighted with a red box. This is where your deployment name and cloud id information are displayed.
Click on the copy button(blue box) then paste the info into line 3 of the default.json
file.
Your default.json
file should look like the following screenshot with your credentials pasted into line 3 and line 5.
Step 4: Connect to Elastic Cloud using basic authentication
Within the server
directory, create a elasticsearch
directory. Within the elasticsearch
directory, create a client.js
file as shown below.
At this point, you might be wondering how we will get our server to communicate with Elastic Cloud so we can establish a secure connection?
It's actually quite simple!
Elasticsearch Client is the simplest and easiest way to communicate with your Elasticsearch cluster from your app, in a programming language of your choice.
Elasticsearch Client
allows you to easily access every endpoint of Elasticsearch API to perform a variety of tasks.
Since we are building a server using Node.js with Express, we will be using the Elasticsearch Javascript Client to connect to our Elasticsearch cluster!
In client.js
, we will create an instance of Elasticsearch Client
. Elasticsearch Client
will contain our Elastic Cloud access credentials that point to our Elasticsearch cluster.
Then, we will set up our server to connect to Elastic Cloud and display a message in the terminal of its connection status.
In the client.js
file, copy and paste the following code.
//in server/elasticsearch/client.js
const { Client } = require('@elastic/elasticsearch');
const config = require('config');
const elasticConfig = config.get('elastic');
const client = new Client({
cloud: {
id: elasticConfig.cloudID,
},
auth: {
username: elasticConfig.username,
password: elasticConfig.password
},
});
client.ping()
.then(response => console.log("You are connected to Elasticsearch!"))
.catch(error => console.error("Elasticsearch is not connected."))
module.exports = client;
Let's go over the code!
Heads up!
For reference purposes only, I have included screenshots of code that I will be explaining.
If you need to copy and paste the code, please refer to the code above or the GitHub repo for connecting to Elastic Cloud with basic authentication.
We require @elastic/elasticsearch
library we installed in part 2 and set this equal to a constant Client
.
This library is the official Node.js client for Elasticsearch and it comes with a lot of functionalities. You will see this coming into play in line 6 when we create a new instance of Elasticsearch Client
.
Line 2
We create a constant called config and require the config
dependency we installed in part 2. You will see this come into play in line 4 when you use this dependency to get access credentials stored in the config
directory.
Line 4
The access credentials were stored under 'elastic'
in config/default.json
as shown below.
We use config
to get
credentials stored under 'elastic'
and set it equal to a constant elasticConfig
.
This allows us to securely store and access sensitive data and not store these in the source code.
Lines 6-14
We create a new instance of Elasticsearch Client
with our access credentials(cloudID, username, and password) that points to our Elasticsearch cluster. We set this equal to a constant called client
.
We use the ping()
method on our Elasticsearch Client
to check if the Elasticsearch cluster is up and available to process requests.
We have it set up so that if we receive a response(meaning we are connected to Elasticsearch), our terminal will display the message "You are connected to Elasticsearch!".
If an error occurs during this process, our terminal should display the message "Elasticsearch is not connected".
Line 20
We expose our Elasticsearch Client
via Node.js module exports as this will be used in server.js
.
Go to the server.js
file within the server
directory. In line 2, add the following snippet.
//in server/server.js
const client = require('./elasticsearch/client');
Your server.js
should look like the following screenshot.
In line 2, we require client.js
from the elasticsearch
directory. We created an instance of Elasticsearch Client
in this file.
We do this as we will need to access our Elasticsearch Client
from server.js
.
Let's test if we are able to connect to Elastic Cloud as a user(basic authentication)!
Start your server by running the following command in the terminal.
//in terminal earthquake_app directory
npm start
You will see that the server is listening on port 3001 and that you are connected to Elasticsearch!
Step 5: Generate an API key
Now that we have logged into Elastic Cloud as a user, we can generate an API key.
From this point on, the API key will be used to connect to Elastic Cloud.
Within the server
directory, create a file called create-api-key.js
.
In create-api-key.js
, copy and paste the following code.
//in server/create-api-key.js
const client = require('./elasticsearch/client');
async function generateApiKeys(opts) {
const body = await client.security.createApiKey({
body: {
name: 'earthquake_app',
role_descriptors: {
earthquakes_example_writer: {
cluster: ['monitor'],
index: [
{
names: ['earthquakes'],
privileges: ['create_index', 'write', 'read', 'manage'],
},
],
},
},
},
});
return Buffer.from(`${body.id}:${body.api_key}`).toString('base64');
}
generateApiKeys()
.then(console.log)
.catch((err) => {
console.error(err);
process.exit(1);
});
Your create-api-key.js
should look like the following.
Let's go over the code!
Heads up!
For reference purposes only, I have included screenshots of code that I will be explaining.
If you need to copy and paste the code, please refer to the code above or the GitHub repo for connecting to Elastic Cloud with API key.
Line 1
We require the instance of Elasticsearch Client
we created in the previous step.
Lines 3-21
We define the generateApiKeys()
function. This function creates an API key that gives access to our Elastic Cloud deployment.
Line 4
We define a constant body
and set it equal to client.security.createAPIKey()
method.
Lines 5-18
Within this method, we define what Elasticsearch cluster privileges are granted to the server connecting with the generated API key.
Line 6
We specify the name of the server using this API key.
Line 7
Under role_descriptors
, we define a role(access privileges) granted to this server.
Line 8
We name this role as earthquakes_example_writer
. You could name this role any way you see fit.
Line 9
We specify that this API key comes with cluster monitor
privilege, which gives read-only access for determining cluster state.
Lines 12-13
This key also grants additional privileges such as create_index
, write
, read
, and manage operations for an index named earthquakes
.
Line 20
The security.createApiKey
function(line 4) returns an id
and api_key
value which are concatenated and encoded in base64
.
Lines 23-28
We call the generateApiKeys()
function here which prints the API key in the terminal. If an error occurs, it prints the error instead.
Let's generate an API key by running the create-api-key.js
file.
In the terminal, create a separate tab. cd
into the earthquake_app directory and run the following command.
//in terminal within the earthquake_app directory
node server/create-api-key.js
You will see the following message in the terminal along with your API key.
Step 6: Connect to Elastic Cloud via API key
We will start by storing the API key in the default.json
file of our config
directory.
Go to default.json
file and add a comma at the end of *line 5. In line 6, copy and paste the following.*
//in config/default.json
"apiKey": "paste your generated API key here"
Copy the API key from your terminal and paste it into *line 6 in default.json
.
Your default.json
file should look like this.
Open the client.js
file in the elasticsearch
directory.
Replace username
and password
(lines 11 and 12) with the following code.
//in elasticsearch/client.js
apiKey: elasticConfig.apiKey
Your client.js
file should now look like the following screenshot(lines 11).
Go back to the terminal tab where you are running the server.
You will see that the server is connected to Elasticsearch except that this time we are connected to Elastic Cloud deployment via the API key!
Now that our server is connected to our Elasticsearch cluster, it is time to think about the data we want to store in Elasticsearch.
Move on to Part 5 to plan for efficient data storage and search performance in Elasticsearch!
Top comments (2)
Hello, I thought you might be interested in wj-config, which is what I think a better configuration package than config.
On the surface it works very similarly to config, but it has a great feature to create URL's out of configuration data that just can't be beaten. Cheers.
I followed exactly what this guide told, but got stuck at step 4: It said "Elastic search is not connected". Can anybody help me? Thanks.