Overview
An OpenSearch cluster requires a cluster manager to be identified in the cluster in order for it to start properly. Furthermore, the election of the cluster manager requires that there be a quorum of 50% and one of the nodes must have voting rights. If the cluster lacks a quorum, it will not start.
Possible causes
Incorrect discovery settings
If you are getting this warning in the logs:
cluster manager not discovered yet this node has not previously joined a bootstrapped cluster
Then the most likely explanation is that you have incorrect settings in opensearch.yml, which prevent the node from correctly discovering its peer nodes.
discovery.seed_hosts: - 192.168.1.1:9300 - 192.168.1.2 - nodes.mycluster.com
The discovery seed hosts should contain a list of nodes in the cluster (of which at least one must be available the first time the node joins the cluster) in order for the discovery function to work.
If it is the first time the cluster has started, then the following setting is also important:
cluster.initial_cluster_manager: - cluster-manager-name1 - cluster-manager-name2 - cluster-manager-name3
Note that here the settings are the node names (not IP addresses) of eligible cluster managers that have the setting:
cluster.manager: true
Cluster manager not elected
If you see the message:
cluster manager not discovered or elected yet, an election requires at least 2 nodes with ids from [UIDIdndidisz99dkhslihn, xkenktjsiasnKKKhdb s, YZ6m2ioDQWqi1cNnOteB6w]
This suggests that a number of cluster managers previously existed in the cluster, but insufficient cluster managers are now available. The likely cause is that cluster managers have been removed from the cluster and so a quorum has not been reached to elect a new cluster manager. If you have recently stopped cluster managers, then you will need to add them back, or wait for OpenSearch to adjust the quorum down to the actual number of cluster manager available.
Node stability issues
If the cluster managers also have the data node role, and are under stress due to heavy indexing or searching, then this can cause them to become unavailable, which in turn can affect the capacity of the cluster to elect a new cluster manager. In clusters that are subject to heavy indexing or search demand, it is recommended to create cluster manager nodes.
If you have other applications running on the same machines as your cluster managers, then it is possible that these processes can leave the cluster manager with insufficient resources to carry out its job, which will result in cluster instability.
In particular if you are running your cluster using containerization technologies such as Docker or Kubernetes, ensure that your cluster managers are guaranteed sufficient resources to be able to do their job and are not destabilized by other processes running on the host machines where the cluster managers are running.
