Overview
There are various “watermark” thresholds on your OpenSearch cluster. As the disk fills up on a node, the first threshold to be crossed will be the “low disk watermark”. The second threshold will then be the “high disk watermark threshold”. Finally, the “disk flood stage” will be reached. Once this threshold is passed, the cluster will then block writing to ALL indices that have one shard (primary or replica) on the node which has passed the watermark. Reads (searches) will still be possible.
Relevant settings
cluster.routing.allocation.disk.watermark have three thresholds of watermarks, it accepts absolute values as well as percentage values. The three watermarks are:
How to fix log messages related to disk watermarks
Permanent fixes
1. Delete unused indices.
2. Merge segments to reduce the size of the shard on the affected node
3. Attach external disk or increase the disk used by the data node
Temporary hacks/fixes
1. Changed these settings values to a higher threshold by dynamically update settings using below update cluster API.
PUT _cluster/settings:
{
“transient”: {
“cluster.routing.allocation.disk.watermark.low”: “100gb”, –>adjust according to your situations
“cluster.routing.allocation.disk.watermark.high”: “50gb”,
“cluster.routing.allocation.disk.watermark.flood_stage”: “10gb”,
“cluster.info.update.interval”: “1m”
}
}2. Disable disk check by hitting below cluster update API
{
“transient”: {
“cluster.routing.allocation.disk.threshold_enabled” : false
}
}Even After all these fixes, OpenSearch won’t bring indices in write mode for that this API needs to be activated
PUT _all/_settings
{
“index.blocks.read_only_allow_delete”: null
}