The Scoop On OpenSearch sizing

Search and analytics means you can search and analyze your data once it has been ingested into OpenSearch.

Getting acquainted with the terminology is mandatory when working with OpenSearch, therefore you should check OpenSearch for Humans - a friendly guide. Additionally, a simple yet concrete OpenSearch Calculator implementation.

Index size? Number of shards per index?

OpenSearch splits indices into shards. Each shard stores a subset of all documents in an index. A shard can be either primary (used for WRITE operations e.g. index, re-index, delete) or replica (used for HA and READ operations e.g. searches). OpenSearch defaults to one primary and one replica shard, for a total of two shards per index.

⚠️ Primary shards cannot be on the same node as the replica.

A shard is a piece of an OpenSearch index, each shard is a full Lucene Index, and each instance of Lucene is a running process that consumes CPU and Memory.

Avoid Oversharding and Node hot spotting

When a shard is involved in an indexing or search request, it uses the CPU to process the request. Each shard you add to an index distributes the processing of requests for that index across an additional CPU. The number of active shards that your domain can support depends on the number of CPUs in the cluster.

Node hot spotting occurs when resource utilizations are unevenly distributed across nodes, e.g. uneven JVM heap size usage. To quickly detect node hot spotting use OpenSearch API:

GET _cat/nodes?v=true&h=name,heap.current,heap.percent

As a rule of thumb, the allocated heap size should be based on the available RAM: set Xms and Xmx to the same value, and no more than 50% of your total memory. A larger Java heap size is useful for indexing, but as memory usage increases, garbage collection becomes more frequent and takes longer.

Shard Count

• Shard count is secondary to shard size. Shard size matters because it impacts both search latency and write performance. 

• A small set of large shards uses fewer resources than many small shards (too many small shards will exhaust the memory - JVM Heap), however, on the other side, too few large shards prevent OpenSearch from properly distributing requests.

• For fast indexing (ingestion), you need as many shards as possible; for fast searching, it is better to have as few shards as possible

When an index is created, the number of shards must be specified and cannot be changed later without reindexing the data. The number of shards you set for an index should correspond to the size of an index, e.g. looking at the two indices store.sizes (one which has replicas set to 0 and the other has replicas set to 1) we can observe that each replica is a full copy of a primary shard.

store.size is the store size taken by primary and replica shards.
pri.store.size is the store size taken only by primary shards.

Total number of shards

First, try to estimate the total size of the data you plan to store in the index and the retention period, and then you can calculate the total number of shards using the formula:

⚠️ Ideally, shard sizes should be between 10GB and 50GB per shard, 10–30 GB for workloads that prioritize low latency (e.g., search workloads), or 30–50 GB (e.g. logging workloads).

Number of shards per index

To prevent hot nodes, OpenSearch distributes shards to instances based on count, where each instance receives as nearly as possible the same number of shards. 

Use shard counts that are multiples of the data node count to ensure that each index is distributed evenly across data nodes. To ensure an even distribution of shards across the data nodes follow the formula:

i.e. If your cluster has 4 nodes and you want to distribute the shards across all nodes evenly, your index should have 8 shards. In other words, you should have at least one shard per data node.

Tunning cluster performance: Search (read) or Ingest (write)?

Search Intensive

• Try to reduce the number of shards as much as possible.
• Replicas improve search performance, so you might want more if you have a read-heavy workload.

Ingest Intensive

• Try to have as many shards, as possible.
• Each replica duplicates the indexing process (new documents are first indexed on the primary and then on any replicas) if you anticipate heavy indexing you can temporarily set the replica count value index.number_of_replicas to 0.
• Increase Index refresh frequency: Indexing documents initially place them into a memory buffer. At this stage, the documents are not yet searchable. To make these documents searchable, a refresh operation is required. OpenSearch refreshes indexes that have received at least one search request in the last 30 seconds, every 1 second.

# GET /_cluster/settings?include_defaults=true
default": {
        "index": {
          "refresh_interval": "1s"
        }
      },

This means that documents written to an active index should typically become searchable within 1 second of being written to OpenSearch. This setting can be adjusted on a per-index basis.

Keep in mind a shorter refresh interval allows documents to become searchable more rapidly post-indexing, but it does so at the expense of increased resource utilization.

Last but not least, some key takeaways: Seven rules for OpenSearch sizing

Links:

• OpenSearch for Humans
• OpenSearch: Do Some Stats on Your Indices
• Tooling for benchmarking: OpenSearch BenchMark for gathering performance metrics and rally a framework for ElasticSearch
• OpenSearch calculator