Elasticsearch is different

Introduction

I'll warn you now, I may sound like an Elasticsearch salesperson in this article but I can assure you that I'm completely impartial.

Until six months ago, I'd used Elasticsearch little more basic document retrieval on simple search terms. I almost resented it as another query "language" to learn. That all changed when I moved into a position which used ES extensively and I needed to get up to speed quickly.

I read the book Relevant Search (https://www.manning.com/books/relevant-search) in two weeks. A superb read which really highlighted the difference between basic document retrieval and real relevant search. I began to see the real power in Elasticsearch. It fired up my imagination in to ways I could have used it with great effect in previous projects.

I wished I'd taken some time to understand the features it offers before now and I believe it could be very valuable to many developers to at least have an overview which is what I hope to give here. It may enable you to make a better decision on software selection.

Notable features

Search engine

Ok, so it's a search engine but your company doesn't require a "search engine". You use some database queries to your NoSQL/relational database which does a decent job?

That's what I thought however the key here is relevance. It's about getting the user where they want/need to be as quickly and as simply as possible. Yes, you can write a database query which puts some data in front of a user which roughly relates to what they put in the search box. And you can tinker with query but it soon becomes very complex and hard to maintain.

Consider search results specific to a user weighted by their previous searches, their favourites, their age, their gender. I don't envy you having to write that SQL query!

Elasticsearch will allow you to query on any of your indexed fields. You can filter your results to create a context for your search. For example, if you the user had specifically said they are searching for a book in your multi-department store, you would filter much as you would in the WHERE clause of a SQL statement.

The good stuff comes when you start telling Elasticsearch that the result may have a certain term or that term but one term should be boosted above the other by a certain factor. You may want to boost more highly rated products or maybe you have some stock that you need to move, that could be boosted in the results.

If you had a library of articles, you may want the most recent ones to be boosted above older articles. If your results were location based, you could boost on distance from the user.

I hope this starts giving you an idea of what relevance is and how a search engine could be useful after all. But that's just the start...

Auto-suggest

ES has built in functionality for auto-suggest. When the user is typing in the search box, you can preempt their request and put it in front of them as they type. Once again, this may have been something you could do with your current database.

ES can be "smarter" though. You can build your indexes so that it looks in any part of words or phrases in multiple fields. If indexed well with good thought, ES can return its results very quickly and minimal overhead on the server.

Aggregation and Facets

ES can return aggregation results in the same result set as the query results. This alone is an appealing feature but it really makes sense. When searching for something, if a user can't find what they were looking for in the results returned, they want an easy step to filter their result set further. Consider I've just searched for TVs on Amazon and it returns thousands of results, how do I get closer to what I want? Facet filters - these are the properties that Amazon display on the left hand side of their desktop page. It allows me to filter further on brand, size, price range etc. ES can easily return these facets and counts for the result set returned.

Aggregations follow what you'd expect from any other database and allowing a pipeline to really dig in to your results.

More like this

A very simple feature to implement but really powerful. It does exactly as it suggests. It allows you to point to a document (or more) and tell ES that you want other results similar to the document you've suggested. This can be very useful for a user to potentially continue their journey on your site or an alternative product similar to the one they're viewing in a shop.

Kibana and visualisations

Kibana is a web UI allowing you to easily build visualisations for your data. It's relatively simple to use and really helps you dig deep in to your data.

It offers all the standard graphing types but also some quite imaginative alternatives. All of these can be put together in dashboards which can be shared and displayed in other pages or on a big screen.

Further reading

I've only touched the surface on an overview of Elasticsearch but I recommend following up with this slightly more in depth overview -https://www.elastic.co/blog/found-uses-of-elasticsearch

Summary

To make it clear, while I make several comparisons to databases, I'm not suggesting that Elasticsearch should replace your database however in many cases I believe it can complement your database.

It could open up functionality to your product that you maybe didn't consider possible. It's very, very fast so can really improve your user experience in places and potentially reduce the load on your database server.

Even if you can't think of a use case within your products, it could potentially open up a whole now world to your logs and analysis on your product performance. With the ability to analyse your data in time series data and dig deep, it could really revolutionise the way you look at your product.