OpenSearch is the flexible, scalable, open-source way to build solutions for data-intensive applications — at least, that's what their website says. So let's delve into its practical use cases: the what, why, and how of OpenSearch.
Let's First Answer Some FAQs
What does OpenSearch do? It's a search and analytics engine — essentially used when you need a search-engine-like tool on top of your database. It can also act as an analytics engine, analyzing large amounts of data, logging, alerting, monitoring, integrating with other BI tools, and basically serving as the backbone of your data storage architecture.
Is OpenSearch the same thing as Elasticsearch? No. OpenSearch was forked from Elasticsearch, but the two have diverged significantly since 2018, and can't be classified as the same tool.
Understanding Keywords
- Document — the basic unit of data in OpenSearch. Think of it like a page you want to store.
- Index — like a table in a relational database, storing where things are. Think of it as a folder with multiple files inside.
- Mapping — another name for your database schema. Think of it as a table of contents: it tells you which document is where, how many pages it has, and which other documents in the folder it's connected to.
- Node — a single instance of OpenSearch. In simple words, a box of files.
- Cluster — multiple nodes working together, or a cabinet full of boxes.
- Shard — similar to a partition in other databases. OpenSearch divides an index into multiple shards and distributes data across different nodes. Imagine three different cabinets of boxes, with documents stored across boxes throughout the cabinets.
- Replica — exactly what it sounds like: a replica of the original documents. OpenSearch stores photocopies of all documents, just in case you lose some.
- Query — like search: shouting at your assistant to find a specific document for you, fast and accurately.
Overview of the Architecture
What is an inverted index?
Inverted indexing maps terms to their corresponding documents. Instead of going through each document to find a term, we go through an inverted index that tells us where a given term is located, and how many times it appears. If a document contains the same term in more than one place, the inverted index records each occurrence.
Data Modeling
Data modeling for your database depends solely on your use case. Here are a few common use cases for an OpenSearch system.
Document Model
This is the core model of OpenSearch, where every data point is a self-contained JSON document with a flexible schema. This kind of data model suits unstructured and semi-structured data, like logs, web pages, and emails. Document modeling allows efficient indexing and retrieval based on keywords and fields within the document, and is the most basic level of data modeling in OpenSearch.
Object Types
If a document can be grouped into a parent-child relationship, an object-type data model makes sense. This is useful for representing complex data structures with parts and subparts — like products with categories and variants — and it also enables efficient searches that navigate through the object hierarchy.
Storing Different Versions of Data
Imagine a document that holds your payslip. Every year, you get an updated payslip reflecting your raise, and you'll want to store all your older payslips alongside the current one. This architecture helps you store different, older versions of a document.
Given that you're storing older data, this requires more storage and adds a bit more complexity to queries — but inserting data becomes much easier. Depending on how often you retrieve historic data, you can also use cold storage.
Storing Up-to-the-Moment Data
Banking systems are the best example of up-to-the-moment data modeling — the moment money is deducted, your bank account should reflect it. Compared to the last scenario, storage requirements are lower here, but inserting data becomes more complex and expensive in some scenarios. To keep an up-to-the-moment data store highly optimized, you need to assess and remove outdated, obsolete data frequently.
Nested Documents
Nested documents let you embed other documents within them, adding a layer of depth. This data model is ideal for data with repeating substructures, like orders with nested order items, and facilitates querying and analysis of both the main document and its nested components.
Denormalization
Denormalization redundantly stores frequently accessed data across multiple documents to improve query performance. It avoids joins and fetches data from a single location, though this requires a strong trade-off between redundancy and data consistency.
Vector Modeling
Vector modeling is a comparatively newer approach, where data points are translated into mathematical vectors for similarity searches. It's useful for finding similar documents based on semantic meaning, like product recommendations or text analysis — to fully understand it, you'll want to dig into statistics and mathematics.
Choosing the right data modeling type depends on your specific data characteristics, queries, and performance requirements. OpenSearch's flexibility allows you to combine these techniques for a customized data model that fits your needs.
Tips to Design a High-Performance System in OpenSearch
Getting the right estimate of storage needs is an important step. As a rule of thumb, use the following equation:
💡 minimum storage required = sourceData × (1 + number of replicas) × 1.45
Here, sourceData is the maximum data to be stored, excluding replicas. The number of replicas in a standard system is usually 2.
Deciding the number of shards is also a major decision, since you cannot change the number of shards once an index is created. To change it, the entire database has to be re-indexed — an expensive operation.
Ideally, 10–30GB is recommended for a read-heavy workload, while 30–50GB is a good number for a write-heavy architecture. Approximating the primary shard size:
💡 Primary Shard = (SourceData + room to grow) × (1 + index overhead) / Desired Shard Size
For an OpenSearch architecture with batched reads and writes, the system typically starts with a 1MB bulk request size, then increases up to 3–5MB per request.
Another important fact to keep in mind: updates are more expensive operations than inserts, so make sure you're using OpenSearch for the right reasons.
OpenSearch, an "Eventually Consistent System"
Since OpenSearch has a distributed architecture with asynchronous replication, it's important to note that data changes aren't instantly propagated to all nodes. This allows for faster writes, but also increases the probability of temporary inconsistencies. OpenSearch refreshes its indices periodically, at a default interval of one second, which can be adjusted.
If you have specific concerns about eventual consistency for your use case, it's recommended to consult the OpenSearch documentation and consider alternative solutions if strong consistency is crucial for your application.
I hope this explanation helps — happy learning!
Co-author credits: Avinash Kedlaya. This article is inspired by, and is a beginner's summary of, this IBM blog post.