JobId not refreshing anomaly detector memory as node is shutting down – How to solve this Elasticsearch error

Opster Team

Aug-23, Version: 8.3-8.9

Briefly, this error occurs when Elasticsearch is trying to refresh the memory of an anomaly detector while the node is in the process of shutting down. This could be due to a manual shutdown or an unexpected failure. To resolve this issue, ensure that the node is fully operational before running the job. If the problem persists, consider increasing the node’s memory or reducing the size of the job. Additionally, check for any network issues that might be causing the node to shut down unexpectedly.

This guide will help you check for common problems that cause the log ” [” + jobId + “] not refreshing anomaly detector memory as node is shutting down ” to appear. To understand the issues related to this log, read the explanation below about the following Elasticsearch concepts: plugin, memory, node.

Log Context

Log “[” + jobId + “] not refreshing anomaly detector memory as node is shutting down” classname is MlMemoryTracker.java.
We extracted the following from Elasticsearch source code for those seeking an in-depth context :

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
// The phaser prevents searches being started after the memory tracker's stop() method has returned
// Note: `phase` is incremented if cache is reset via the feature reset API
if (stopPhaser.register() != phase.get()) {
// Phases above not equal to `phase` mean we've been stopped; so don't do any operations that involve external interaction
stopPhaser.arriveAndDeregister();
logger.info(() -> "[" + jobId + "] not refreshing anomaly detector memory as node is shutting down");
listener.onFailure(new EsRejectedExecutionException("Couldn't run ML memory update - node is shutting down"));
return;
}
ActionListener<long> phaserListener = ActionListener.wrap(r -> {
stopPhaser.arriveAndDeregister();
</long>
// The phaser prevents searches being started after the memory tracker's stop() method has returned // Note: `phase` is incremented if cache is reset via the feature reset API if (stopPhaser.register() != phase.get()) { // Phases above not equal to `phase` mean we've been stopped; so don't do any operations that involve external interaction stopPhaser.arriveAndDeregister(); logger.info(() -> "[" + jobId + "] not refreshing anomaly detector memory as node is shutting down"); listener.onFailure(new EsRejectedExecutionException("Couldn't run ML memory update - node is shutting down")); return; } ActionListener<long> phaserListener = ActionListener.wrap(r -> { stopPhaser.arriveAndDeregister(); </long>
        // The phaser prevents searches being started after the memory tracker's stop() method has returned
        // Note: `phase` is incremented if cache is reset via the feature reset API
        if (stopPhaser.register() != phase.get()) {
            // Phases above not equal to `phase` mean we've been stopped; so don't do any operations that involve external interaction
            stopPhaser.arriveAndDeregister();
            logger.info(() -> "[" + jobId + "] not refreshing anomaly detector memory as node is shutting down");
            listener.onFailure(new EsRejectedExecutionException("Couldn't run ML memory update - node is shutting down"));
            return;
        }
        ActionListener phaserListener = ActionListener.wrap(r -> {
            stopPhaser.arriveAndDeregister();

 

 [ratemypost]