Beginner's guide to understanding the relevance of your search with Elasticsearch and Kibana

“I love it when my search engine retrieves irrelevant search results,” said nobody ever.

As developers, we search for things on a daily basis. Whether we are searching for a solution on how to fix a bug or how to build a new feature, it is incredibly frustrating when you are not quite getting what you are looking for.

That is what relevance is all about!

When you search for something on your app, you want to get results that are directly related to what you are looking for.

Elasticsearch is a powerful search and analytics engine designed to do just that! By integrating Elasticsearch into your app, you can create a great search experience for your users.

This blog will help you understand the relevance of your search with Elasticsearch and Kibana!

By the end of this blog, you will be able to:

1) learn how precision and recall are used to measure how well Elasticsearch is searching for user's query
2) understand how scoring is used to rank relevancy of search results in Elasticsearch
3) use resources provided at the end of the blog to master Elasticsearch queries that fine-tunes the precision or recall of your search results

Additional resources

Interested in beginner friendly workshops on Elasticsearch and Kibana? Check out my Beginner's Crash Course to Elastic Stack series!

Here is the table of content of all the workshops I have created for the series thus far. This repo contains all resources shared during the workshops including the video recording, presentation, related blogs, Elasticsearch queries and more!

How do we measure relevance in Elasticsearch?

There are multiple factors that come into play. This blog will focus on precision and recall.

Before we get to that, let's do a quick recap!

Elasticsearch is a search engine that allows us to store, search, and analyze data. It stores data as documents. Documents that share similar traits are grouped into an index.

When you search for something, Elasticsearch retrieves relevant documents(green boxes). Then, it presents these documents as search results(orange box).

The two diagrams in the image above depict the same thing.

The diagram on the left shows documents grouped into an index. The same thing is shown on the diagram on the right.

The yellow rectangle represents your sample index. The grey dots are the documents contained in that index.

This blog will use the diagram on the right to explain precision and recall. But before we do that, let’s go over some terms!

What are true positives?

When you send a search query to Elasticsearch, it retrieves documents that it considers relevant to the query.

These are the dots inside the white circle. These are the documents that Elasticsearch sends back as a response.

Some of the these retrieved documents are what you expect to see in your response. These are known as true positives.

What are false positives?

You probably had an experience where you searched for something and some of the results were not relevant to what you were looking for.

These are known as false positives. These are irrelevant search results that were retrieved by the Search engine for some reason.

What are true negatives?

Let’s focus on the dots in the yellow region of this diagram. These are the documents that were not returned by the search engine.

Some of them are truly irrelevant to the search query and were correctly rejected by the search engine. These are known as true negatives.

What are false negatives?

Among the rejected documents, there may be relevant documents that should have been returned in the response. These are known as false negatives.

Now that we have mastered these terms, let's get back to relevance! Earlier, I have mentioned that precision and recall are used to measure the relevance of a search engine.

Precision

The term precision has to do with the dots inside the white circle. These are documents that are returned as search results.

Precision is calculated by true positives divided by the sum of true positives and false positives. Precision tells you what portion of the retrieved data is actually relevant to the search query.

Recall

Recall on the other hand is calculated by true positives divided by the sum of true positives and false negatives. It tells you what portion of relevant data is being returned as search results.

Precision and recall are inversely related.

Precision wants all the retrieved results to a perfect match to the query, even if it means returning less or no documents.

Whereas recall focuses more on quantity. It wants to retrieve more results even if the documents may not be a perfect match to the query. The dilemma is that we want to present the really relevant items but we also want to retrieve as many results as possible.

As you can see, these two factors are at odds with each other because if you want to improve precision, it might cause a decline in recall and vice versa.

Let’s recap real quick! Precision and recall determine which documents are included in the search results. However, precision and recall do not determine which of these returned documents are more relevant than the other.

This is determined by ranking!

When you look at your search results, you will see that the most relevant results are at the top and the least relevant are at the bottom.

This ranking or order is determined by a scoring algorithm.

Each result is given a score. Ones with the highest scores are displayed at the top whereas ones with the lowest scores are displayed at the bottom.

What is a score?

Score is a value that represents how relevant a document is to that specific query. A score is computed for each document that is a hit, and hits are search results that are sent to the user.

The higher the score a document has, more relevant the document is to the query, and it’s going to end up higher in the order!

How is a score calculated?

There are multiple factors that are used to compute a document's score. This blog will only focus on term frequency(TF) and inverse document frequency(IDF).

When you search for something, you type in a search query in the search box. Elasticsearch looks at the query and pulls up relevant documents or hits.Then, it calculates a score for each document and ranks them by relevance.

So how does this happen?

What is term frequency?

Let’s talk about how term frequency plays a role in calculating a score.

In the image below, we have a search query(red bracket) “ how to form good habits”. This query is made of multiple search terms(blue arrows).

When the relevant documents are retrieved, Elasticsearch looks through these documents and calculates how many times each search term appears in a document.

This is known as term frequency.

If a document mentions search terms more frequently, Elasticsearch assumes that this document is more relevant to the search query and it assigns a higher score to the document.

Let’s say we were looking at the frequency for the term habits(orange box).

In the field description of the first document(red box), the term habits appears 4 times(TF=4). In the field description of the the second document(blue box), the term habits appears 1 time(TF=1).

Therefore, in this example, the first document will be given a higher score and end up higher on the search results.

What is Inverse Document Frequency(IDF)?

When we calculate a score based on term frequency alone, this will not give us the most relevant documents. This happens because term frequency considers all search terms to be equally important when assessing the relevance of a document.

Let’s look at the search query in the image below.

We have search terms, "how" and "to" and "form" and "good" and "habits". Not all of these search terms will help you determine the relevance of a document.

For example, the first four search terms are commonly seen in many, if not all documents. Take a look at the hits(blue box), then at the documents highlighted with an orange box.

The documents like "how to form a meetup group" or "good chicken recipes" do contain some of the search terms. But these documents are completely irrelevant to what we are looking for!

But because of term frequency, if these commonly found search terms were found in high frequency in any of these documents, these documents will end up with high scores, even though these are irrelevant to the query.

So Elasticsearch offsets this with inverse document frequency(IDF). With Elasticsearch, if certain search terms are found in many documents in the result set, it knows that these terms are not useful at determining relevance.

When Elasticsearch goes through all the hits, it will reduce the score for documents with unimportant search terms. It will also increase the score for documents with important search term like habits.

Fine-tuning precision and recall via Elasticsearch queries

Now that we have covered the basics of relevance, we are ready to fine-tune the precision or recall of our search results! The best way to show you how these queries work is by sending these queries from Kibana to Elasticsearch.

There are three things you must complete in order to follow along! This repo contains all of the directions and resources you will need to complete these steps.

When you open the repo and scroll down to the Resources section, you will see the following.

Step 1: Set up Elasticsearch and Kibana
You can access Elasticsearch and Kibana in two ways:
1) Access Elasticsearch and Kibana hosted on Elastic Cloud
2) Download Elasticsearch and Kibana on your local machine

Instructions for each option is included under Resources(green box).

Option 1: Accessing Elasticsearch and Kibana hosted on Elastic Cloud

Unlike the downloaded option, Elastic Cloud handles all the heavy lifting of managing the stack so you can focus on building your product instead.

You get a free 30 day trial. There is no credit card required and the trial expires on its own. I find that for beginners it is easier and faster to get started on Elastic Cloud.

Option 2: Downloading Elasticsearch and Kibana on your local machine

This option is free and there is no expiration date. However, you will need to manage Elasticsearch and Kibana on your own.

Both options will perfectly be able to handle all of the queries we are about to go over. Please choose whatever option that works best for you!

Step 2: Add data to Elasticsearch

From the same repo, click on the video of the workshop(orange box). Go to time stamp 18:57.

This video will walk you through how you can add the dataset(yellow box) to Elasticsearch.

Step 3: Master the queries to fine-tune the precision or recall of your search results

The rest of the video will walk you through different queries that allows you to fine-tune the precision and recall of your search results!

All the search queries covered in the video are included in the same repo(below the Resources section). So make sure to keep the repo open while you follow along.

All right, you are all set to go. Master these queries and explore what you can do with Elasticsearch and Kibana!