Performing CRUD operations with Elasticsearch & Kibana

In the Beginner's Guide to Elasticsearch, we have mastered the basic architecture of Elasticsearch. We also got a chance to download and run Elasticsearch and Kibana on our local machine as well.

Now that we have mastered the basics of these two products, it is time to get hands-on experience with Elasticsearch and Kibana!

Elasticsearch is a powerful search and analytics engine. It is known as the heart of the Elastic Stack. Elasticsearch stores, searches, and analyzes your data.

Elasticsearch stores data as documents. This blog will teach you how to perform CRUD operations using Elasticsearch and Kibana.

By the end of this blog, you will be able to Create, Read, Update, and Delete a document from Elasticsearch.

Let's get started!

Resources

If this is your first time learning about Elasticsearch, check out the following resources before getting started with this blog. This blog builds on the content shared in these resources.

1) Beginner's Guide to Elasticsearch

2) Instructions for downloading and running Elasticsearch and Kibana

3) Video of Beginner's Crash Course to the Elastic Stack

For those of you who prefer to learn via video, I have created a recording that covers the content shared in the Beginner's Guide to Elasticsearch in detail.

This video also contains a hands-on lab section where I cover how to perform CRUD operations with Elasticsearch and Kibana(25:03).

Be sure to check out the description of this video as it includes the timestamps of each sections as well as resources shared during the workshop.

Prerequisite Work

Follow these instructions to complete the following steps.

1. Download Elasticsearch and Kibana.

2. Run Elasticsearch and Kibana.

3. Load Kibana console on a browser.
   You should see the following on your screen.
   

To follow along, have two windows open side by side. As shown below, you will have the Kibana console open on the left and this blog open on the right.

You are now ready to perform CRUD operations! Using Kibana, we will send requests to Elasticsearch to Create, Read, Update, and Delete a document.

Performing CRUD operations

Background info

In Elasticsearch, data is stored as documents. A document is a JSON object that stores whatever data you want to store in Elasticsearch. Each document has a unique ID.

Let's say that you are storing documents about goods sold at an online grocery store. A document for one grocery item would look like the following.

In a JSON object, you have a list of fields or key value pairs. For example, it has the name of the product, category it belongs in, its brand and price.

Things are much easier to find when you group them in a logical manner!

Documents that are logically related to each other are grouped into an index. For example, documents of carrots and clementines would be grouped under the produce index. Documents of Malbec and IPAs would be grouped under the wine and beer index.

For this exercise, we are going to perform CRUD operations on documents about our favorite candy.

Let's start by creating an index!

C - Create

Create an index

Syntax:

PUT Name-of-the-Index

To create an index, we use the HTTP verb PUT followed by the name of the index.

Since we are indexing documents about our favorite candy, we will name our index favorite_candy as shown in the following example.

Example:

PUT favorite_candy

Go to your Kibana console.

The console is divided into two panels. The left panel is where we send requests to Elasticsearch. The right panel is where we get a response back from Elasticsearch.

Delete the default query. Copy and paste the following request into the left panel(line 1).

PUT favorite_candy

In order to send the request, the first line of the request must always be selected. We do that by clicking on the request, which should put a gray bar over the request. Click on the arrow(blue box) to send the request.

In the right panel of the Kibana console, you will see the following response from Elasticsearch.

You will see a 200-OK response returned to you. When you see "acknowledged": true(line 2), that means that an index called favorite_candy(line 4) has been successfully created.

Index a document

Now that we have an index, let's index some documents!

We use the word index as both a noun and a verb. When index is used as a verb, it means that we are storing documents in Elasticsearch.

When indexing a document, either POST or PUT can be used.

1) POST
You use POST when you want Elasticsearch to autogenerate an id for your document.

Syntax:

POST Name-of-the-Index/_doc
{
  "field": "value"
}

We start the request with a POST HTTP verb, then we specify the name of the index, then the document endpoint(_doc), followed by a JSON object with whatever data you want to store in Elasticsearch.

In our example, I am going to index a document about my favorite candy. When this syntax is applied to our example, it looks like the following.

Example:

POST favorite_candy/_doc
{
  "first_name": "Lisa",
  "candy": "Sour Skittles"
}

This POST request directs Elasticsearch to index the following document(_doc) in the favorite_candy index. This document has a field named first_name, which has a value of Lisa. It also has a field named candy, which has a value of Sour Skittles. Since the verb POST is used, Elasticsearch knows to autogenerate an id for this document.

Copy and paste this request into the Kibana console(lines 3-7). Make sure the first line(line 3) of the request is selected and send the request.

You will get the following response from Elasticsearch.

You will see a 201-Created response(green box). This response states that in our index called favorite_candy(line 2), we have a document(_doc, line 3) with an autogenerated id(line 4) that has been created(line 6).

Great job! You have just indexed your first document!

Earlier, I have mentioned that you can either use POST or PUT verb when indexing a document.

2) PUT
You use the verb PUT when you want to assign a specific id to your document. An instance where you may want to use PUT is when you are indexing data with a natural identifier(i.e. purchase order number, patient id, & etc).

For example, let's say you are indexing patient data where each patient has a unique ID. At this point, sending POST request is the only way you know how to index documents, which autogenerates document IDs.

This method will work just fine until you need to look up patient info by the document ID.

These are randomly generated IDs(ex. "_id" : "EoZTjHYBnFpZV1kVcyTF") that have no meaning or relations to the data it stores. How will you keep track of these ids of assorted letters and numbers? Will you have to keep track of all of these IDs in addition to the list of patient data containing patient IDs?

If only there was an easier way to index and find these documents...

There is a better way!

Using PUT, you can assign the patient ID as the ID for the patient's document. This way, you can use the patient ID to find the document that you are looking for!

If you want to learn more about using POST vs PUT to index documents, check out this documentation from Elastic.

Let's get back to indexing more documents. The following syntax is used when using PUT to index data.

Syntax:

PUT Name-of-the-Index/_doc/id-you-want-to-assign-to-this-document
{
  "field": "value"
}

When this syntax is applied to our example, it looks like the following.

Example:

PUT favorite_candy/_doc/1
{
  "first_name": "John",
  "candy": "Starburst"
}

This request is directing Elasticsearch to PUT in favorite_candy index the following document(_doc), the JSON object in the next lines. It also directs Elasticsearch to assign this document an id of 1.

Copy and paste this request into the Kibana console(lines 9-13). Make sure to select the first line of the request(line 9) and send the request.

Elasticsearch will send back the following response.

You will see a 201-Created response(green box). This response states that in the index favorite_candy(line 2), a document(_doc, line 3) with an assigned id of 1(line 4) has been created(line 6).

Note the "_version":1 in line 5. We will go over this in a bit!

R - READ

Read a document

Now that we have indexed a document, let's send a request to examine the content of the document that has been indexed.

We use the following syntax to read a document.

Syntax:

GET Name-of-the-Index/_doc/id-of-the-document-you-want-to-retrieve

When we apply this syntax to our example, it looks like the following.

Example:

GET favorite_candy/_doc/1

This request directs Elasticsearch to GET from favorite_candy index a document(_doc) with an id of 1.

Copy and paste this request into the Kibana console(line 15). Make sure the request is selected and send the request.

You should see the following response from Elasticsearch.

You will see a 200-OK response(green box). In the JSON object, you will see that in an index called favorite_candy(line 2), there is a document(_doc, line 3) with an assigned id of 1(line 4). In the _source field(line 9), you will see the content of document 1 (lines 10 and 11).

As you can see, reading a document is a great way to check whether our CRUD operations have been successfully completed.

What do you think would happen if we accidentally index another document with an ID that already exists?

Let's find out.

Copy and paste the following into the Kibana console(lines 15-19), right below the request for indexing document 1 that contains John's information.

PUT favorite_candy/_doc/1
{
  "first_name": "Sally",
  "candy": "Snickers"
}

Notice that we are indexing Sally's document with an ID that already exists(red boxes)!

Make sure to select the first line(line 15) of this request and send the request.

You should get the following response from Elasticsearch.

Notice that we are getting a 200-OK response instead of 201-OK response this time. It tells you that a document(_doc, line 3) with an id of 1(line 4) has been updated(line 6).

Note that version number has now been changed to 2 (line 5). Why is that happening?

_version number signifies the number of times the document has been created, updated, or deleted. Since document 1 has been originally created with John's favorite candy information then accidentally updated with Sally's info, the version number is now 2.

Let's double check the result of this request by sending a GET request we have just used(line 21).

Select and send the GET request. You will see the following response.

If you look at lines 10-11, you will see that John's information has been overwritten by Sally's information!

Uh oh... That is not good. We do not want to accidentally overwrite an existing document.

To prevent this from happening, we can use the _create endpoint!

_create Endpoint

_create endpoint prevents you from overwriting an existing document. When this endpoint is in use, the request to index a document with an existing ID is denied. Elasticsearch will throw an error and reject the request.

The following syntax is used when _create endpoint is used.

Syntax:

PUT Name-of-the-Index/_create/id-you-want-to-assign-to-this-document
{
  "field": "value"
}

When this syntax is applied to our example, it looks like the following.

Example:

PUT favorite_candy/_create/1
{
  "first_name": "Finn",
  "candy": "Jolly Ranchers"
}

This request directs Elasticsearch to PUT in index favorite_candy the following JSON object and assign it an id of 1. HOWEVER(_create), if a document with ID of 1 exists, then do not index this document. Throw an error message instead.

Copy and paste the request into the Kibana console(lines 23-27), right below the GET request. Make sure the first line of the request(line 23) is selected and send the request.

You should see the following response from Elasticsearch.

Elasticsearch sends a 409-Conflict error. It states the reason as document already exists(lines 6-7).

Let's send a GET request used in the previous example to make sure nothing has been changed.

Select the GET request we used earlier(line 21) and send the request.

You will see that Sally's information is still there (lines 10-11) and nothing has changed!

As you can see, the _create endpoint provides a safeguard for you so you do not accidentally overwrite an existing document.

Let's move on to update!

U - UPDATE

Update a document

There will be times when you will want to update an existing document. For example, let's say Sally originally liked Snickers but her favorite candy now is M&Ms.

You will use the following syntax to update a field of a document.

Syntax:

POST Name-of-the-Index/_update/id-of-the-document-you-want-to-update
{
  "doc": {
    "field1": "value",
    "field2": "value",
  }
} 

In the JSON object, make sure to add "doc". This tells Elasticsearch that only the fields specified in the nested curly brackets are meant to be updated.

When this syntax is applied to our example, it looks like the following.

Example:

POST favorite_candy/_update/1
{
  "doc": {
    "candy": "M&M's"
  }
}

This POST request directs Elasticsearch to update(_update) a document with an id of 1 in the favorite_candy index. By including "doc" in the JSON object, it also clarifies that only the field and value specified in nested curly brackets should be updated. The rest of the information in the original document should be kept in tact.

Copy and paste this request into the Kibana console(lines 21-26), right below the request to index Sally's document. Select the first line(line 21) of the request and send the request.

You should see the following response from Elasticsearch.

You will see a 200-OK response from Elasticsearch. The response states that a document(_doc, line 3) with an id of 1(line 4) has been updated(line 6). Notice that _version number has been changed to 3(line 5).

This makes sense because _version keeps track of number of times a document has been created, updated, and deleted. Document 1 has been originally created with John's information(_version: 1). It was accidentally overwritten by Sally's information(_version: 2). Then, we intentionally updated the field candy with M&M's(_version:3).

Let's send a GET request to check the content of document 1. Select the previous GET request we have used and send the request (line 28).

You will see that the field candy has been updated with M&M's(neon green box)!

Last but not least, let's delete a document!

D- DELETE

Delete a document

The following syntax is used to delete a document.

Syntax:

DELETE Name-of-the-Index/_doc/id-of-the-document-you-want-to-delete

When we apply the syntax to our example, it looks like the following.

Example:

DELETE favorite_candy/_doc/1

This request directs Elasticsearch to DELETE from favorite_candy index a document(_doc) with an ID of 1.

Copy and paste this request into the Kibana console(line 28), right below the update candy field request.

You will see the following response from Elasticsearch.

You will see a 200-OK response that states that document(_doc, line 3) with an id of 1(line 4) has been deleted(line 6). The version number is now 4(line 5).

If you send a GET request for document 1 again, you will get a 404 error message because document 1 no longer exists!

There you have it. You are now the master of performing CRUD operations with Elasticsearch and Kibana.

Put your newly learned skills to use and try the following exercise on your own!

Take Home Assignment

1. Create an index called places.
2. Pick five of the places you want to visit after the pandemic is over. For each place, index a document containing the name and the country.
3. Read(GET) each document to check the content of the document.
4. Update a field of a document.
5. Read(GET) the updated document to ensure that the field has been updated.
6. Delete a document of one place.
7. Copy and paste the following request to return all documents from the places index. This is a great way to check whether all the CRUD operations you have performed thus far have worked! ```

GET places/_search
{
"query": {
"match_all": {}
}
}