Elasticsearch Index Templating in Elasticsearch – How to Use Composable Templates

By Opster Expert Team - Madhusudhan

Updated: Jan 28, 2024

| 6 min read

Quick links

The source code for this article is available here.

Overview

An Elasticsearch index can be configured through mapping, settings and aliases: 

  • Mapping definitions specify the data schema.
  • Settings will set the shard sizing and refresh rates. 
  • Aliases are used to give the index alternate names. 

When we index a document for the first time or create an empty index using the Create Index API, the index will be created with default settings, without data schema and without aliases. These defaults work pretty well in development and testing environments, but we may need to customize our indices for production environments.

Working with the default mappings and settings in production might result in poor indexing and search performance. Instantiating indices manually is a tedious and time consuming process. Recreating such indices on every environment is especially impractical if we have an elaborate mapping schema as well as customized settings and aliases.

Fortunately, Elasticsearch provides us with a tool to automatically apply a predefined configuration when creating indices in the form of index templates.

Index templates

Index templates allow us to create indices with user defined configuration. An index can pull the configuration from these templates, for example a set number of shards and replicas or field mappings, during its instantiation. A template will be defined with a name pattern and some configuration in it. If the name of the index matches the template’s naming pattern, the new index will be created with the configuration defined in the template.

Elasticsearch has upgraded its template functionality in version 7.8 with composable templates. This newer version provides for much more reusable index templates, as demonstrated  in this article.

Types of templates

Index templates can be classified into two categories: 

  • Index templates (or composabale index templates): The composable index templates can either exist on their own, or can be composed of none or more component templates (see the second category). 
  • Component templates: The component template is a reusable template on its own that defines the required configuration. Usually the component template is expected to be associated with an index template. Each of the component templates can be attached with one or many index templates. 

As you can see in the image below, the index templates A and B share component templates (in this case just one – Template 3) between themselves. An index template can consist of none or many component templates and each of the component templates can be associated with none or many index templates. Both types of templates can exist on their own, however component templates are of no use unless they are attached to an index template.

The general idea is to develop a catalogue of component templates for an organization to use for various needs (for example, specifying the various component templates for individual environments) and attach them to various indices via the composable index templates. 

How to create composable (index) templates

Elasticsearch provides an _index_template endpoint for managing index templates. The user provides all the required mappings, settings, and aliases along with an index name pattern in this template. Let’s run through an example of creating a template for a microservice application customer-order-service which is responsible for order generation logic. 

Let’s say our requirement is to create a template for customer orders, represented with a pattern having wildcards: *orders. This template is expected to have certain mappings  and settings, such as the order_date field, as well as shards and replica numbers. 

Any index that gets matched with this template during its creation inherits the configurations defined in this template. For example a black_friday_orders index will have the order_date field, shards will be set to 5 and the replicas set to 2. In addition to this, all indices created from this template inherit a single alias name, too!Let’s create this orders_template with an index pattern defined as *orders and with a mapping schema consisting of a single oder_date field with a predefined date format dd-MM-yyyy. The code below shows how to create this index template.

PUT _index_template/orders_template
{
  "index_patterns": ["*orders"],
  "priority": 300,
  "template": {
    "mappings": {
      "properties": {
        "order_date": {
          "type": "date",
          "format":"dd-MM-yyyy"
        }
      }
    },
    "settings":{
      "number_of_shards":5,
      "number_of_replicas":2
    },
    "aliases":{
      "all_orders":{}
    }
  }
}

When you execute this query in Kibana’s DevTools, the template is created with the index pattern *orders along with the predefined mapping, settings and an alias. The index_patterns is an array of match patterns; any index matching this pattern will be deriving the template configuration. You can execute the following to retrieve the persisted template that should reiterate what we did:

GET _index_template/orders_template

There’s also a priority, a positive number, defined when creating the template attribute defined on the template: every template is defined with a priority so that any conflicting changes from different templates will be resolved by using this value with precedence given to the higher priority value. We’ll dive more deeply into template priority below. 

Creating an index with the template 

Now we have a template – a blueprint for creating indices – the next step is to create an index. When the name of the index matches with the given pattern, the templated configurations are applied automatically. To prove the point, as the code below shows, let’s create a brand new index named: blackfriday_orders:

PUT blackfriday_orders

As the name of the index (blackfriday_orders) matches with the naming pattern defined in template (i.e. *orders), the index should get all the configuration derived from the template. Let’s retrieve this freshly created index and check if this is indeed true by executing the following code:

GET blackfriday_orders

This should return:

{
  "blackfriday_orders" : {
    "aliases" : {
      "all_orders" : { }
    },
    "mappings" : {
      "properties" : {
        "order_date" : {
          "type" : "date",
          "format" : "dd-MM-yyyy"
        }
      }
    },
    "settings" : {
      "index" : {
         ...
        "number_of_shards" : "5",
        "number_of_replicas" : "2"
      }
    }
  }
}

As the response indicates, the configuration of the blackfriday_orders has been inherited from the template. We can try with various combinations of the indices that will successfully inherit the templated configuration:

PUT blackfriday_orders
PUT americaorders
PUT cancelled--orders
PUT undefined101orders

However, the following indices will not be inheriting the configuration as the name will not match with the pattern:

PUT blackfriday_orders2
PUT open_orders_
PUT allorders_total

One important thing to remember is that all the indices derived from a template have the same alias – all_orders – in this case. There is an advantage of having such alias – we can simply query on this single alias rather than multiple indices.

GET blackfriday_orders,americaorders,undefined101orders/_search
GET all_orders/_search 
{
  "query": {
    "range": {
      "order_date": {
        "gte": "01-12-2021",
        "lte": "31-12-2021"
      }
    }
  }
}

While we create a template for *orders, any matching index is expected to adopt the template configuration. Usually, knowingly or unknowingly, teams may create a few more templates for various reasons. This means that at times the index name may match two different template patterns! Elasticsearch has to decide which of the configurations from those templates it needs to apply. Fortunately, this dilemma can be solved by using the template priority. 

Creating component templates

We learned about index templates in the earlier part of this article. There are a couple of disadvantages of creating the templates with the configuration built in – one such disadvantage being that the configuration is not exportable for other templates. If we wish to have similar configuration, say for customer related templates (*customers), we may have to re-create the whole template. That means, we may be creating dozens of them in a typical organization (plus you may have a few more based on the environments). 

As we always look forward to reusability, Elasticsearch redesigned the templates keeping reusability in mind. The component templates fit that bill. If you are from a DevOps background, most likely you’d have a requirement to create indices with a preset configuration for each of the environments. Rather than tediously applying each of these configurations manually, you can create a component template for each of the environments. 

A component template is nothing but a reusable block of configurations that we can use to make up more index templates. Do note that the component templates are of no value unless they are clubbed with index templates. They are exposed via a _component_template endpoint. Let’s see how this all fits together.

Settings template

Let’s extract the settings we defined in our index template earlier and create a component template out of it. The settings_component_template is expected to have five primary shards with two replicas per primary shard. The first step, as the code listing below shows, is to declare and execute a component template with this configuration.

PUT _component_template/settings_component_template
{
  "template":{
    "settings":{
      "number_of_shards":5,
      "number_of_replicas":2
    }
  }
}

As the code above shows, we use the _component_template endpoint to create a component template. The body of the request holds the template information in a template object. The settings_component_template is now available for use elsewhere in the index templates. One notable difference is that this template does not define any index pattern; it’s simply a code block that configures some properties for us. 

Mappings template

In the same way, let’s create another template. This time, let’s extract the mapping schema we had defined earlier in the standalone index templates. The code below shows the script:

PUT _component_template/mappings_component_template
{
  "template": {
    "mappings": {
      "properties": {
        "order_date": {
          "type": "date",
          "format":"dd-MM-yyyy"
        }
      }
    }
  }
}

Aliases template

Going with the same flow, we can also have a component template with the aliases – two aliases (all_orders and sales_orders):

PUT _component_template/aliases_component_template
{
  "template": {
    "aliases": {
      "all_orders": {},
      "sales_orders":{}
    }
  }
}

Composable index template

Now that we have these three component templates, the next step is to put them to use. We can do this by letting an index template for, say christmas_orders, use it:

PUT _index_template/composed_orders_template
{
  "index_patterns": [
    "*orders"
  ],
  "priority": 500,
  "composed_of": [
    "settings_component_template",
    "mappings_component_template",
    "aliases_component_template"
  ]
}

The composed_of  tag is a collection of all the component templates that make up this template. In this case, we are choosing the settings, mappings and aliases component templates. We are also bumping up the priority so this template trumps any others. Once the template is ready, any indices that match the *orders pattern will inherit the configuration from these three component templates.

Having said that, should we wish to create a new template, say customers, with just one of the existing (settings_component_template) and a newly created aliases (aliases_component_template – see below) template, we can do so with:

PUT _component_template/aliases_component_template2
{
  "template": {
    "aliases": {
      "all_customers": {}
    }
  }
}

The index template goes like this:

PUT _index_template/composed_customers_template
{
  "index_patterns": [
    "*customers*"
  ],
  "priority": 200,
  "composed_of": [
    "settings_component_template",
    "aliases_component_template2"
  ]
}

Did you see that the settings_component_template has been (re)used across two different templates? That is the power of component templates.

Template priority     

There is a chance that developers may create multiple index templates without looking at the existing stock. It is important to set a priority on each of these templates so the one with higher priority will be used. For example, the my_orders_template_1 overrides the my_orders_template_2 in the following code snippet:

PUT _index_template/my_orders_template_1
{
  "index_patterns": ["*orders"],
  "priority": 1000,
  "template": { ... }
}
PUT _index_template/my_orders_template2
{
  "index_patterns": ["*orders"],
  "priority": 300,
  "template": { ... }
}

When you have multiple templates matching the indices that are being created, Elasticsearch applies all the configurations from all matching templates but overrides anything that has higher priority.

Precedence of templates

Finally, you may be wondering about the precedence of the templates – does the configuration defined in the component template override the one defined on the main index template itself? Or the other way around? Well, there are some rules:

  • An index created with configurations explicitly takes precedence over everything – this means if you create an index with explicit configuration, don’t expect them to be overridden by the templates.
  • Legacy templates (templates created before version 7.8) carry a lower priority than the composable templates.

Summary

  • An index contains mappings, settings and aliases: the mappings define the fields schema, settings set the index parameters such as number of shards and replicas, and aliases give alternate names to the index.
  • Templates allow us to create indices with predefined configurations. Naming an index with a name that matches the index-pattern defined in a specific template will automatically configure that index according to the template.
  • Elasticsearch introduced composable index templates in version 7.8. Composable index templates allow modularity and versioning of templates. 
  • The composable templates consist of none or more component templates. 
  • An index template can have its own configuration defined too. 
  • A component template is a reusable template with predefined configuration, just like a composable index template. 
  • However, component templates are expected to be part of an index template; They are useless if they are not “composed” into an index template. 
  • Component templates have no index pattern defined in them – which is another reason they are “expected” to be part of an index template.
  • Each of the templates has a priority – a positive number. The higher the number, the greater the precedence for that template to be applied.

Related Articles

Back to all articles