Quick Links
- Overview and background
- What is the “cluster allocation explain API” used for and how can it be used?
- Analyzing common outputs returned by the cluster allocation explain API
Overview and background
Elasticsearch is a clustered technology, which stores data in primary shards and replica shards. Each shard must be allocated to one of the servers forming the Elasticsearch cluster.
An index in Elasticsearch consists of one or multiple shards grouped together creating an index. Every single shard serves as a self-contained index. To guard against hardware failures and boost query capacity, Elasticsearch can provide redundancy when nodes are added to a cluster.
Redundancy involves distributing the documents in an index across several shards and distributing those shards across several nodes. Elasticsearch automatically migrates shards to rebalance the cluster as it expands (or shrinks).
Elasticsearch has two algorithms: shard allocation, which determines which unallocated shards should belong to which nodes, and shard rebalancing, which involves shifting a shard from one node to another.
When your cluster shows a “yellow” status, it means that Elasticsearch has not been able to allocate one or more replica shards. A “red” cluster indicates that one or more primary shards have not been allocated, meaning queries may return inaccurate or incomplete data, or you may lose data since Elasticsearch cannot index or update records.
Definition & Uses
The “cluster allocation explain API” is a very useful way to determine either why a given shard is unassigned or why it did not relocate in the way you were expecting it to.
The “cluster allocation explain API” is a very useful way to determine either why a given shard is unassigned or why it did not relocate in the way you were expecting it to.
To be able to use this API, you must have the manage cluster or monitor privilege if Elasticsearch security features are enabled. The following requests are used to explain cluster allocation:
GET _cluster/allocation/explain
POST _cluster/allocation/explain
If you call the API without any POST data, then Elasticsearch will retrieve an allocation explanation for a random unassigned replica or primary shard. The API gives a 400 error if there are no unassigned shards in the cluster.
The cluster allocation explain API has the following query parameters:
- include_disk_info: an optional parameter of type “Boolean.” When it is set to true, the response will show the shard sizes and the disk usage details. The default value is false.
- include_yes_decisions: an optional parameter of type “Boolean.” When it is set to true, the YES decisions will be returned in the explanation. The default value is false.
If you want an explanation for a specific node or shard, then the cluster allocation explain API has the following request body parameters:
- current_node: an optional parameter of type “string” that specifies the node name or ID to explain a shard that is presently located on the particular node.
- index: an optional parameter of type “string” that specifies the name of the index for which you need the explanation.
- primary: an optional parameter of type “Boolean.” When set to true, it returns the primary shard’s explanation for the specified shard ID.
shard: an optional parameter of type “integer” that represents the ID of the shard for which you want an explanation.
Analyzing common outputs returned by the cluster allocation explain API
The request below is used to get the allocation explanation for the unassigned primary shard with the ID “5” in the “my-index” index:
GET _cluster/allocation/explain
{
"index": "my-index",
"shard": 5,
"primary": true
}And it will create an explanation response as follows:
{
"index" : "my-index",
"shard" : 5,
"primary" : true,
"current_state" : "the shard's current state",
"unassigned_info" : {
"reason" : "the cause of the shard being unassigned.",
"at" : "2022-01-04T19:09:16.600Z",
"last_allocation_status" : "no"
},
"can_allocate" : "whether or not to assign the shard",
"allocate_explanation" : "Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster. Choose a node to which you expect this shard to be allocated, find this node in the node-by-node explanation, and address the reasons which prevent Elasticsearch from allocating this shard there.",
"node_allocation_decisions" : [
{
"node_id" : "9bt2rY-pT6KNZB5-aBcw",
"node_name" : "node-1",
"transport_address" : "125.0.0.2:9401",
"node_attributes" : {},
"node_decision" : "whether or not to assign the shard to the specific node",
"weight_ranking" : 1,
"deciders" : [
{
"decider" : "the decider that decides the decision for the node",
"decision" : "the decision",
"explanation" : "an explanation for the decider's "no" decision, with a useful indication of the setting that caused the choice"
}
]
}
]
}The most common issues that cause unassigned shards are:
- The disk is full.
- The node containing the data has relocated to a different host.
- Shards cannot be assigned according to the current allocation rules.
- The allocation of searchable snapshot indices is prevented by a snapshot issue.
- The maximum number of retries has been exceeded.
- The maximum number of shards per node would be exceeded.
- The number of replicas exceeds the number of eligible data nodes (leading to the unassigned replica shards problem).
These issues are discussed in more detail below.
Disk space issues
When you get one or more of the messages below in the cluster allocation explain API’s response, it indicates that disk use has exceeded the limit:
- the node is above the high watermark cluster setting [cluster.routing.allocation.disk.watermark.high=90%], using more disk space than the maximum allowed [90.0%], actual free: [9.273781776428223%].
- unable to force allocate shard to [%s] during replacement, as allocating to this node would cause disk usage to exceed 100%% ([%s] bytes above available disk space).
- the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85%], using more disk space than the maximum allowed [85.0%], actual free: [14.119771122932434%].
- after allocating [[restored-xxx][0], node[null], [P], recovery_source[snapshot recovery [Om66xSJqTw2raoNyKxsNWg] from xxx/W5Yea4QuR2yyZ4iM44fumg], s[UNASSIGNED], unassigned_info[[reason=NEW_INDEX_RESTORED], at[2022-03-02T10:56:58.210Z], delayed=false, details[restore_source[xxx]],allocation_status[fetching_shard_da a]]] node [GTXrECDRRmGkkAnB48hPqw] would have more than the allowed 10% free disk threshold (8.7% free), preventing allocation.
The quickest way to solve disk space issues is to DELETE any unnecessary indices. If that is not an option, you must consider increasing the disk size of the nodes (if possible) or adding new data nodes. Another option is to increase the disk watermark settings, but bear in mind that this is only a short-term way of buying time before finding a long-term solution. For a full discussion of disk watermark issues, and their solutions, please see our guide on the subject.
Missing primary shards
When you have no available shard replicas and a node is lost, you typically receive this message:
- cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster.
In this case, you will need to recover the node where the primary shard was last seen, or the shard in the index will be permanently lost.Hot cold allocation rules
When you get one or more of the messages below in the cluster allocation explain API’s response, it is a sign that shards cannot be assigned based on the current attribute-based or data tier allocation rules:
- node does not match index setting [index.routing.allocation.include] filters [node_type:\”cold\”].
- index has a preference for tiers [data_cold] and node does not meet the required [data_cold] tier.
- index has a preference for tiers [data_cold,data_warm,data_hot] and node does not meet the required [data_cold] tier.
- index has a preference for tiers [data_warm,data_hot] and node does not meet the required [data_warm] tier.
- this node’s data roles are exactly [data_frozen] so it may only hold shards from frozen searchable snapshots, but this index is not a frozen searchable snapshot.
Note that the word “preference” is misleading in the above messages because Elasticsearch treats a “preference” as a hard requirement.
In this case, you must either add new nodes with the required node_type or change the roles of existing nodes to be of the required node_type by modifying the setting in elasticsearch.yml:
node.roles: [ data_warm, data_cold ]
If you want to remove the setting from the index in order to allow the index to allocate shards on a different node type, use the following:
PUT my-index/_settings { "index.routing.allocation.include._tier_preference": null }For a full explanation of hot-cold shard allocation, see our article on the subject here.
Snapshot issues
When a snapshot issue, such as permission problems or missing data, is preventing the allocation of searchable snapshot indices, you will get a message similar to the one below in the response of the cluster allocation explain API:
- failed shard on node [Yc_Jbf73QVSVYSqZT8HPlA]: failed recovery, failure RecoveryFailedException[[restored-my_index-2021.32][1]: … SnapshotMissingException[[found-snapshots:2021.08.25-my_index-2021.32-default_policy-_j2k8it9qnehe1t-2k0u6a/iOAoyjWLTyytKkW3_wF1jw] is missing]; nested: NoSuchFileException[Blob object [snapshots/52bc3ae2030a4df8ab10559d1720a13c/indices/WRlkKDuPSLW__M56E8qbfA/1/snap-iOAoyjWLTyytKkW3_wF1jw.dat] not found: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 4AMTM1XFMTV5F00V; S3 Extended Request ID:
Node max shards setting
The overall number of primary and replica shards in the cluster is constrained by the “cluster.max_shards_per_node” setting. If your cluster has more shards than allowed, the following message may appear:
- Validation Failed: 1: this action would add [2] shards, but this cluster currently has [1000]/[1000] maximum normal shards open
To solve this issue, you should ideally reduce the number of shards on your cluster by deleting unnecessary data. As a short-term solution, you can temporarily increase the limit by running:PUT /_cluster/settings { "transient": { "cluster.routing.allocation.total_shards_per_node": 1100 } }However, you should try to find a better longer-term solution to reduce the number of shards. You can read more on this here or here.
Retry issues
Before giving up and leaving a shard unallocated, the cluster will make several attempts to assign it. Index.allocation.max_retries on Elasticsearch Service is set by default to five. You may receive the following message if allocation fails after the maximum number of tries:
- shard has exceeded the maximum number of retries [%d] on failed allocation attempts – manually call [/_cluster/reroute?retry_failed=true] to retry, [%s]
As the message above states, if you believe that there is no reason for the allocation to fail, you can manually retry by running the following:POST/_cluster/reroute?retry_failed=true
However, if the issue persists, then read carefully the results of the allocation explain API, which will explain the reason why the allocation is failing.
- failed shard on node [Yc_Jbf73QVSVYSqZT8HPlA]: failed recovery, failure RecoveryFailedException[[restored-my_index-2021.32][1]: … SnapshotMissingException[[found-snapshots:2021.08.25-my_index-2021.32-default_policy-_j2k8it9qnehe1t-2k0u6a/iOAoyjWLTyytKkW3_wF1jw] is missing]; nested: NoSuchFileException[Blob object [snapshots/52bc3ae2030a4df8ab10559d1720a13c/indices/WRlkKDuPSLW__M56E8qbfA/1/snap-iOAoyjWLTyytKkW3_wF1jw.dat] not found: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 4AMTM1XFMTV5F00V; S3 Extended Request ID:
