Quick link
- What is document-level alerting in OpenSearch?
- How to use document-level alerting
- How to use query-based alerts in Elasticsearch
- Conclusion
OpenSearch document-level alerting: definition
Document-level alerting is a new OpenSearch feature that allows activities to be detected not only based on a query or an aggregation but also against documents at the moment they are indexed. So, instead of running an alert detection on a schedule, one or more queries are run against the document when it is ingested.
Query and bucket monitors can only run a single query with one trigger condition. However, document-based monitors allow you to combine multiple query triggers via tags.
When you do that, all the queries are evaluated in an OR operation. So, if any of the queries are met, an alert will be triggered.
When an alert is triggered, a document is added to the “findings” index. It contains the document ID and the query that triggered the alert, thus allowing users to run audit operations against this meta index.
In this article, we will show you how to use document-level alerting in OpenSearch and how query-based alerts in Elasticsearch can achieve similar results.
NOTE: To create alerts in OpenSearch, you must have at least one Channel created. To learn how to set up an alerting Channel, you can read our article on the subject: OpenSearch Alerting – How to Set Up Alerting in OpenSearch (opster.com)
How to use document-level alerting
Let’s look at an example scenario to understand how to use document-level alerting in OpenSearch.
In this scenario, we want to be notified every time a bitcoin sell action transaction is made.
Step 1. Create the index:
PUT test_transactions
{
"mappings": {
"properties": {
"action": {
"type": "keyword"
},
"currency": {
"type": "keyword"
},
"amount": {
"type": "long"
}
}
}
}Step 2: Configure the alert. The way to do that is to go to Alerting -> Create monitor.
Step 3. Set the basic details. We want to create a per document monitor and use the visual editor for the queries:
When setting the details, the key parts are the queries:
And the triggers:
Step 4. Ingest some documents:
POST test_transactions/_doc
{
"action": "sell",
"amount": 100,
"currency": "BTC",
}POST test_transactions/_doc
{
"action": "sell",
"amount": 200,
"currency": "CLP"
}After some time, we will see the alert triggered by the first document we sent and the findings index for further analysis:
Step 5: We must mark the alert as acknowledged to stop executing.
This is what the finding looks like:
And this is the alert we receive:
Monitor documents just entered alert status. Please investigate the issue. - Trigger: WH - Severity: 1 - Period start: 2022-12-02T22:36:38.947738922Z - Period end: 2022-12-02T22:36:38.947740964Z
How to use query-based alerts in Elasticsearch
The way to achieve something similar with Elasticsearch is to use a query-based alert via Kibana Alert Rules. The limitation is a timestamp field that must be provided because the approach is not tracking the document but a query over time.
Step 1: Create the same index we created for OpenSearch in the example above:
PUT test_transactions
{
"mappings": {
"properties": {
"action": {
"type": "keyword"
},
"currency": {
"type": "keyword"
},
"amount": {
"type": "long"
}
}
}
}Step 2. Add the @timestamp field:
POST test_transactions/_doc
{
"@timestamp": "2022-12-01T13:16:40.139Z",
"action": "sell",
"amount": 100,
"currency": "BTC"
}POST test_transactions/_doc
{
"@timestamp": "2022-12-01T13:16:40.139Z",
"action": "sell",
"amount": 200,
"currency": "CLP"
}Step 3. Set up an index query rule:
Step 4: Add the following query:
{
"query": {
"bool": {
"filter": [
{
"term": {
"action": "sell"
}
},
{
"term": {
"currency": "BTC"
}
}
]
}
}
}Step 5: If we want to send one alert per finding, it is important to mark “Exclude matches from previous runs”.
After some time, you will see the alert triggered:
Conclusion
Using document-level alerts, we can run one or more queries against a document at the moment that it is indexed and trigger an alert if it meets the conditions. This feature is very useful when we need to react to a document with specific field values, instead of an aggregation or the output of a query.
Elasticsearch offers a different functionality via Kibana Alert Rules, which is focused on a group of documents (queries) and not an individual document.
