Why index lifecycle management is necessary
Index lifecycle management is a feature that helps automate the creation, management and deletion of an Elasticsearch index.
Being able to automate the creation of a new index when the index reaches the optimum size of 50GB per shard is very useful. Setting up a time-based index with one index per day, or one index per month, is likely to create index shards that have an optimal size. Shards that are either too small or too large can cause various issues- see this guide for more info.
Once an index is no longer being written to, it is useful to optimize the index to save cluster resources. This can involve:
- Force merging the index to optimize the space used by the index on disk.
- Shrinking the index to reduce the number of shards.
- Allocating the shard to less performant / storage optimized hardware.
As an index nears the end of its life we may want to:
Watch the video below on the importance of setting up ILM in your deployment.
How to set up index lifecycle management
Define lifecycle management policy
An index lifecycle management policy can be applied to any number of indices. The policy consists of a number of defined phases and indicates the actions that should be taken when transitioning from one phase to the next.
In Elasticsearch’s multi-tier implementation, the tiers are named hot, warm, cold and frozen. And, throughout its lifecycle, data traverses them in that order. It’s not required to implement all of them, only the hot tier is present in most use cases, since it serves as an entry point – all policies must start with a HOT phase.
The transition from one phase to the next is defined by the time passed since the creation of the index. For more information on hot-warm-cold architecture, see the Multi-Tier Architecture Bible. The diagram below shows an example of multi-tier architecture.
PUT _ilm/policy/hot_warm_delete
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "50gb",
"max_age": "3d"
},
"set_priority": {
"priority": 50
}
}
},
"warm": {
"min_age": "7d",
"actions": {
"forcemerge": {
"max_num_segments": 1
},
"set_priority": {
"priority": 25
},
"shrink": {
"number_of_shards": 1
}
}
},
"delete": {
"min_age": "15d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}The above policy defines the following:
Hot phase
During the hot phase, a new index will be created when the size of the index reaches 50GB or when the index is 3 days old.
Warm phase
After 7 days, the hot index will pass to the warm phase. The index will be force merged and a shrink operation will be carried out to reduce the index to a single shard.
Delete phase
After 15 days, the index will be deleted.
Note: In this case no “cold” or “frozen” phases have been configured.
Setting up a template to add the ILM Policy to the index settings
PUT _index_template/my_apache_log
{
"index_patterns": ["my_apache_log*"],
"template": {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "hot_warm_delete",
"index.lifecycle.rollover_alias": "my_apache_log"
}
}
}The above command creates an index template to apply the hot_warm_delete policy we created above to any indices with the prefix “my_apache_log”.
It also defines the alias that will be applied to the new index when the original index rolls over.
Manually create the first index
We need to manually create the first index for the my_apache_log series.
PUT my_apache_log-000001
{
"aliases": {
"my_apache_log": {
"is_write_index": true
}
}
}The above index creates the first index with the my_apache_log alias.
We then need to make sure the client application sends all writes to the alias “my_apache_log” (not the full index name). Elasticsearch will write these logs to index my_apache_log-000001 until that index reaches a size of 50GB (as defined in our ILM policy) and then roll over the index, automatically creating a new index my_apache_log-000002 and so on.
Modifying the ILM policy
If you modify the ILM policy, the modifications will be applied to all of the indices that use that policy. However the changes will only be applied for the new changes of phase. Any changes of phase that have already been applied with previous versions of the policy cannot be rolled back.
ILM actions
The most common actions are rollover, force merge, wait for snapshot and delete. However all of the following actions are possible:
- Allocate settings to the index to move the shards to different nodes, or change the number of replicas.
- Delete the index.
- Force merge the index to optimize structure on disk.
- Freeze the index to minimize its memory footprint. This is only appropriate for old indices which are very rarely searched. Note that a frozen index will take seconds or even minutes to search since certain data structures will need to be rebuilt on the fly.
- Migrate index shards to the data tier being used by the ILM phase. This is only appropriate if you are using data tiers in your architecture.
- Read only to prevent writing to the index.
- Rollover to create a new write index.
- Searchable snapshot the index to enable the use of this mechanism to reduce memory and disk requirements for infrequently searched indices.
- Set priority of an index so that Elasticsearch knows which indices to recover first.
- Shrink the number of primary shards in an index (this creates a new index).
- Unfollow a cross cluster replication index. This is only appropriate for clusters using Cross Cluster Replication.
- Wait for snapshot before deleting index.
ILM errors
If you see that ILM is not behaving as expected, you can check the status of ILM on an index using the following command:
GET /my-index-000001/_ilm/explain
Read the error message carefully and try to fix the error. Typically the error may be corrected, either by modifying the ILM policy, or by correcting the index settings. Once you have done this, you can run this command:
POST /my-index-000001/_ilm/retry
