Elasticsearch Elasticsearch Hot Warm

By Opster Team

Updated: Mar 24, 2024

| 2 min read

Introduction

A hot-warm architecture is a popular approach to optimize Elasticsearch clusters for both indexing and searching performance. This architecture is designed to handle time-based data, such as logs and metrics, which are frequently indexed and searched. In this article, we will discuss the benefits of using a hot-warm architecture and provide a step-by-step guide to setting up a hot-warm Elasticsearch cluster. If you want to learn about Elasticsearch multi-tier architecture – how to set up a hot/warm/cold/frozen Elasticsearch architecture, check out this guide.

Benefits of Hot-Warm Architecture

1. Improved performance: By separating indexing and searching operations, hot-warm architecture allows for better resource allocation and improved overall performance.

2. Cost-effective: Hot-warm architecture enables the use of different hardware profiles for hot and warm nodes, reducing overall infrastructure costs.

3. Scalability: This architecture allows for easy scaling of the cluster by adding more hot or warm nodes as needed.

Setting Up a Hot-Warm Elasticsearch Cluster

Step 1: Configure node roles

First, configure the Elasticsearch nodes with the appropriate roles. Hot nodes should be assigned the “data_hot” role, while warm nodes should be assigned the “data_warm” role. Update the elasticsearch.yml configuration file on each node with the following settings:

For hot nodes:

node.roles: [ "data_hot", "ingest", "remote_cluster_client" ]

For warm nodes:

node.roles: [ "data_warm" ]

For master nodes:

node.roles: [ "master" ]

Step 2: Implement index lifecycle management (ILM)

To automate the process of moving data from hot to warm nodes, use Elasticsearch’s index lifecycle management (ILM) feature. Create an ILM policy that defines the phases and actions to be taken:

PUT _ilm/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "7d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "allocate": {
            "require": {
              "data": "warm"
            }
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      }
    }
  }
}

Step 3: Apply the ILM policy to the index template

Finally, create an index template that will apply the appropriate settings to new indices. The template should include settings for the number of shards, replicas, the ILM policy and the data tier preference. For example:

PUT _index_template/my_template
{
  "index_patterns": ["my_index-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.routing.allocation.include._tier_preference": "data_hot",
      "index.lifecycle.name": "my_policy"
    }
  }
}

Conclusion

By following these steps, you can set up a hot-warm Elasticsearch cluster that optimizes performance and resource allocation.