Overview
Sometimes you can observe that the CPU and load on some of your data nodes is higher than on others.
This can occasionally be caused by applications that are not load balancing correctly across the data nodes, and are making all their HTTP calls to just one or some of the nodes. You should fix this in your application.
However it is more frequently caused by “hot” indices being located on just a small number of nodes. A typical example of this would be a logging application creating daily indices with just one shard per index. In this case although you may have many indices spread across all of the shards, you may find that all of the indexing is being done on just one shard on one node which contains today’s logging index.
A similar thing could happen at a search level, if for example you have a “products” index which is intensively searched by users.
How to fix it
Check your monitoring data to see which nodes are affected, and look to see which nodes and indices on that node have the highest index and search rates.
Load balance across all data nodes
You should fix this by putting a load balancer in front of your Elasticsearch nodes, or by including ALL of the nodes in the client application.
es = Elasticsearch(
['clientNode1', 'clientNode2','clientNode3'],
http_auth=('user', 'secret'),
scheme="https",
port=443,
)The above example is how you can load balance across 3 nodes (you should include all the nodes) when using the python client without a load balancer. All of the official Elasticsearch clients use similar arrays in their construction.
Hot indexing activity
The indexing activity is carried out by primary shards, so if you have just a few “hot” indices doing all of the indexing activity it may be desirable to increase the number of primary shards in order to spread this activity over several nodes. The number of shards is determined when the index is created, so it is a good idea to control that using an index template. Before creating the template, check to see whether a template already exists for the index pattern using:
GET _cat/templates
Templates are often created to control mappings, but you can also control shards and replicas in the same template. To avoid confusion and conflicts between templates it is often better to create only one template per index. The command below will create or modify the template called “mytemplate_name”:
PUT _index_template/mytemplate_name
{
"index_patterns": ["myindex*"],
"template": {
"settings": {
"number_of_shards": 3
},
"mappings": {
..etc....
}
}
}The change to your template will not take effect until your index rolls over and a new index is created.
Hot search activity
If your application is search intensive rather than index intensive, then you can spread this activity across the nodes by creating extra replicas. Unlike hot indexing activity, with search operations it is NOT necessary to create extra primary shards. Instead you can create extra replicas up to the number of nodes minus 1.
You can change the number of replicas of an existing index using:
PUT /<my index>/_settings
{
"index" : {
"number_of_replicas" : 3
}
}Alternatively, you might like to set this using a template:
PUT _index_template/mytemplate_name
{
"index_patterns": ["myindex*"],
"template": {
"settings": {
"number_of_shards": 1,
"number_of_replicas":3
},
"mappings": {
..etc....
}
}
}