Failed to generate source – How to solve this Elasticsearch exception

Opster Team

August-23, Version: 6.8-8.9

Briefly, this error occurs when Elasticsearch fails to generate a specific source due to issues like incorrect syntax, invalid data type, or missing fields in the source. To resolve this, you can check the syntax of your query for any errors, ensure that the data types are correct, and verify that all required fields are included in the source. Additionally, check if the source exists and is accessible. If the problem persists, consider debugging your code to identify the root cause of the issue.

This guide will help you check for common problems that cause the log ” Failed to generate [” + source + “] ” to appear. To understand the issues related to this log, read the explanation below about the following Elasticsearch concepts: source.

 
 

source

Overview

When a document is sent for indexing, Elasticsearch indexes all the fields in the format of an inverted index, but it also keeps the original JSON document in a special field called _source. 

Examples

Disabling source field in the index:

PUT /api-logs?pretty
{
  "mappings": {
    "_source": {
      "enabled": false
    }
  }
}

Store only selected fields as a part of _source field:

PUT api-logs
{
  "mappings": {
    "_source": {
      "includes": [
        "*.count",
        "error_info.*"
      ],
      "excludes": [
        "error_info.traceback_message"
      ]
    }
  }
}

Including only selected fields using source filtering:

GET api-logs/_search
{
  "query": {
    "match_all": {}
  },
  "_source": {
       "includes": ["api_name","status_code", "*id"]
  }
}

Notes

The source field brings an overhead of extra storage space but serves special purposes such as:

  • Return as a part of the response when a search query is executed.
  • Used for reindexing purpose, update and update_by_query operations.
  • Used for highlighting, if the field is not stored, it means  the field is not set as “store to true” inside the mapping.
  • Allows selection of fields to be returned.

The only concern with source field is the extra storage usage on disk. But this storage space used by source field can be optimized by changing compression level to best_compression. This setting is done using index.codec parameter.

 
 

 
 

Log Context

Log “Failed to generate [” + source + “]” class name is XContentHelper.java. We extracted the following from Elasticsearch source code for those seeking an in-depth context :

 public static XContentParser mapToXContentParser(XContentParserConfiguration config; Map source) {
 try (XContentBuilder builder = XContentFactory.contentBuilder(XContentType.JSON)) {
 builder.map(source);
 return XContentFactory.xContent(builder.contentType()).createParser(config; Strings.toString(builder));
 } catch (IOException e) {
 throw new ElasticsearchGenerationException("Failed to generate [" + source + "]"; e);
 }
 }
}

 

 [ratemypost]

Related Articles

Back to all articles