Elasticsearch Elasticsearch Version Upgrades – Using Feature Migration APIs to Avoid Deprecation Issues

By Opster Team

Updated: Mar 22, 2023

| 3 min read

Introduction

It’s always complicated to upgrade to the new Elasticsearch version and transfer data from the old Elasticsearch cluster to the new one. Before migrating, it’s very important to check which Elasticsearch version and which Elasticsearch index version you are currently using. Knowing the current version of Elasticsearch is the most important item and it is effective in all subsequent processes.

What is the goal of Feature Migration APIs? 

These APIs were created to simplify migration tasks and automate the migration process. The feature migration APIs simplify upgrading X-Pack indices from one version to another. This means that whenever you are upgrading your Elasticsearch cluster to a more recent version, you may be facing these types of issues: 

• Index version issues (index structural aspect)
• Elasticsearch plugins versions
• Snapshot versions (snapshots are more flexible after version 7.17 and 8)
• Deprecated and removed index-level settings
• Deprecated and removed node-level settings
• Deprecated and removed cluster-level settings
• Deprecated and removed configuration-level settings (elasticsearch.yml)

What you need to know before migrating to a new Elasticsearch version

The most important part of migration is data and removed settings at each level of Elasticsearch configurations (Index, Node, Cluster). Beyond this, in most major version updates of Elasticsearch, there are some important changes that affect your data and require you to do some corrections on your data mappings to fit the new version requirements.

For example, if you are currently using Elasticsearch version 7.15 and you want to upgrade to Elasticsearch version 8.2, you should consider revising all the security configuration since there have been some major overhauls. Also, adding new nodes to the cluster is not the same process as it was in previous versions.

How to retrieve information about deprecated settings

Before upgrading Elasticsearch you can use the Deprecation API to retrieve information about all settings at the: 

• Cluster Level
• Node Level 
• Index Level 

When running the following command, you can glean some helpful information about deprecations: 

GET /_migration/deprecations

Output for Elasticsearch Version 7.16: 

Output for Elasticsearch Version 8.2:

There are two level of issues you may facing in response: 

• Warning – You can upgrade directly, but you are using a deprecated functionality which will not be available or will behave differently in a future version.
• Critical – You cannot upgrade without fixing this problem.

You can read more on deprecation here.

How to retrieve information about deprecated settings regarding features

Before doing anything, you should get acquainted with the deprecations concerning the  features you’re using in your current ES cluster version. This information will help you prevent serious problems from happening during your migration.This feature helps you retrieve information about system index deprecations for the next major version in Elasticsearch so be careful to use this data for any changes and any migrations. 

The Feature migration API was added to Elasticsearch 7.17. Prior to that version, there was  only the deprecation info API we saw earlier. 

Similarly to the Deprecation API, the Feature Migration API also shows you the upcoming deprecation of: 

• Cluster-Level settings
• Node-Level settings
• Index-Level settings

Below you can see the output of the system_features API call that shows you the deprecated feature name.

GET /_migration/system_features
Output:
{
  "features" : [
    {"feature_name" : "async_search",
      "migration_status" : "NO_MIGRATION_NEEDED",
    },
    {
      "feature_name" : "enrich",
      "migration_status" : "NO_MIGRATION_NEEDED",
    },
    {
      "feature_name" : "fleet",
      "migration_status" : "NO_MIGRATION_NEEDED",
      ]
    },
    {
      "feature_name" : "geoip",
      "migration_status" : "NO_MIGRATION_NEEDED"
    },
    {
      "feature_name" : "kibana",
      "migration_status" : "NO_MIGRATION_NEEDED"
    },
    {
      "feature_name" : "logstash_management",
      "migration_status" : "NO_MIGRATION_NEEDED"
    },
    {
      "feature_name" : "machine_learning",
      "migration_status" : "NO_MIGRATION_NEEDED"
    },
    {
      "feature_name" : "searchable_snapshots",
      "migration_status" : "NO_MIGRATION_NEEDED"
    },
    {
        "feature_name": "security",
        "minimum_index_version": "6.8.20",
        "migration_status": "MIGRATION_NEEDED",
            "indices": [
                {
                    "index": ".security-6",
                    "version": "6.8.20"
                }
            ]
        }
,
    {
      "feature_name" : "tasks",
      "migration_status" : "NO_MIGRATION_NEEDED"
    },
    {
      "feature_name" : "transform",
      "migration_status" : "NO_MIGRATION_NEEDED"
    },
    {
      "feature_name" : "watcher",
      "migration_status" : "NO_MIGRATION_NEEDED"
    }
  ],
  "migration_status" : "MIGRATION_NEEDED"
}

After running the command you may see that some migration is needed.

Based on the above output you can run the migration process by running the POST command below and it will update the features as needed.

POST /_migration/system_features

In general, this feature is part of the “Kibana Upgrade Assistant” that you can use to upgrade your Elasticsearch cluster to the newer node and solve some system issues automatically: 

Please take a look at the Upgrade Assistant screenshot below which demonstrates how the Feature Migration API can help you through the migration process (Elasticsearch, Kibana version 7.17):

Important notes

Always make sure to take a snapshot of your current data before you start your upgrade!

You cannot downgrade an Elasticsearch cluster after upgrading. If the upgrade process breaks for any reason, you will need to restore data from the snapshot.