OpenSearch API

Coralogix provides an OpenSearch API that allows you to query your hosted OpenSearch instances securely and with ease.

Our OpenSearch API, which supports REST API, requires backward compatibility with Elasticsearch from v6.0 to v7.10. Find more information here.

Query your Coralogix OpenSearch API

To use the OpenSearch API, it is necessary to:

STEP 1. Add a Coralogix authorization key with each HTTP request. You need to create a personal or team API key. It’s recommended to use permission presets, as they are automatically updated with all relevant permissions. Alternatively, you can manually add individual permissions.

Preset	Action	Description
DataQuerying	`LEGACY-ARCHIVE-QUERIES:EXECUTE` `LOGS.DATA-API#HIGH:READDATA` `LOGS.DATA-API#LOW:READDATA` `METRICS.DATA-API#HIGH:READDATA` `SPANS.DATA-API#HIGH:READDATA` `SPANS.DATA-API#LOW:READDATA`	Query Data from the Archive Query Frequent Search Logs Query Monitoring & Compliance Logs Query Metrics Query Frequent Search Spans Query Monitoring & Compliance Spans

STEP 2. Select the OpenSearch API endpoint associated with your Coralogix domain.

STEP 3. Input the index name. If you want to query logs, then use *. If you want to query Logs2Metrics, then use *:*_log_metrics*.

Note: The user may query only legacy Logs2Metrics. New Logs2Metrics can be queried with PromQL only.

URL: <cx_endpoint_url>/<index>/_search

Examples:

Log querying URL: https://api.app.coralogix.in/data/os-api/*/_search

Logs2Metrics querying URL: https://api.app.coralogix.in/data/os-api/*:*_log_metrics*/_search

STEP 4. The API request should contain the following:

Header	Value
Authorization	Bearer `<cx_api_key>`
Content-type	application/json

Examples:

curl "<cx_endpoint_url>/*/_search" -H "Authorization: Bearer <cx_api_key>" -H "Content-type: application/json" -d '{
    "query": {
        "bool": {
            "must":
            [
                {
                    "term": {
                        "coralogix.metadata.applicationName": "PROD"
                    }
                },
                {
                    "range": {
                        "coralogix.timestamp": {
                            "gte": "now-15m",
                            "lt": "now"
                        }
                    }
                }
            ]
        }
    },
    "aggs": {
        "severities": {
            "terms": {
                "field": "coralogix.metadata.severity"
            }
        }
    }
}'

curl "<cx_endpoint_url>/*/_search" -H "Authorization: Bearer <cx_api_key>" -H "Content-type: application/json" -d '{
    "query": {
        "bool": {
            "must":
            [
        {
            "match": {
                "text": "created"
            }
        },
                {
            "range": {
                "coralogix.timestamp": {
                    "gte": "now-15m",
                    "lt": "now"
                    }
                }
             }
            ]
        }
    }
}'

curl "<cx_endpoint_url>/*/_search" -H "Authorization: Bearer <cx_api_key>" -H "Content-type: application/json" -d '{
    "query": {
        "bool": {
            "filter":[{
                "query_string": {
                    "query": "YOUR QUERY"
                }
             },
                {
            "range": {
                "coralogix.timestamp": {
                    "gte": "2019-10-23T14:00:00",
                    "time_zone": "+03:00"
                }
            }
                }
            ]
        }
    }
}'

Using the OpenSearch Scroll API

curl -H "Authorization: Bearer <cx_api_key>" -H "Content-type: application/json" -d '{
    "size": 1000,
    "query": {
        "bool": {
            "filter": [{
                    "query_string": {
                        "query": "YOUR QUERY"
                    }
                },
                {
                    "range": {
                        "coralogix.timestamp": {
                            "gte": "now-24h",
                            "lt": "now"
                        }
                    }
                }
            ]
        }
    }
}'

You will receive the first batch of logs along with a new field in the root of the response named _scroll_id.

That same scroll_id should be used in subsequent requests to create the pagination, and obtain the next log batch. You should repeat the second request until all logs are retrieved.

Second and subsequent scroll API requests, should contain the Headers and URL endpoint. Note that an index is not specified in the endpoint for the second, and subsequent requests.

Example #1:

curl -H "Authorization: Bearer <cx_api_key>" -H "Content-type: application/json" -d '{

    "scroll": "5m",
    "scroll_id": "YOUR_SCROLL_ID"

}'

Example #2:

curl --location --request POST "<cx_endpoint_url>" \
--header "Authorization: Bearer <cx_api_key>" \
--header 'Content-Type: application/json' \
--data-raw '{
    "scroll": "5m",
    "scroll_id": "YOUR_SCROLL_ID"

}'

The last log page retrieved will show and empty array at the bottom of the page:

"hits": []

This indicates that there are no more log pages to retrieve.

Multi-search

curl --location --request POST '<cx_endpoint_url>' \
--header 'Authorization: Bearer <cx_api_key>' \
--header 'Content-Type: application/json' \
--header 'Content-Type: application/x-ndjson' \
--data-raw '{"index":"*"}
{"query":{"bool":{"filter":[{"match_phrase":{"coralogix.metadata.applicationName":{"query":"coralogix"}}},{"match_phrase":{"coralogix.metadata.subsystemName":{"query":"coralogix"}}},{"range":{"coralogix.timestampsdsd":{"gte":"now-2h","lte":"now"}}}]}}}

{"index":"*"}
{"query":{"bool":{"filter":[{"match_phrase":{"coralogix.metadata.applicationName":{"query":"coralogix"}}},{"match_phrase":{"coralogix.metadata.subsystemName":{"query":"coralogix"}}},{"range":{"coralogix.timestampsdsd":{"gte":"now-4h","lte":"now-2h"}}}]}}}'

Every query should be in online format and start with {"index":"*"}

Limitations

The Coralogix OpenSearch API has the following limitations:

Supports only POST requests.
Supported top-level elements of the Search API: query, from, size,
sort, _source, post_filter, aggs, aggregations.
The sum of the top-level elements ‘from' and ‘size' cannot be
greater than 12000.
allow_leading_wildcard element in query_string query is not allowed.
Wildcard queries can’t start with '*' or '?'
RegEx queries can’t start with '.*' or '.?'
max_determinized_states element inside regex queries is not allowed.
Size element for bucket aggregations cannot be greater than 1200.
The bucket aggregation of the type significant_terms is not allowed.
Nesting of the following bucket aggregations 3 or more times is not allowed: date_histogram, geohash_grid, histogram,
ip_ranges, and terms.
fuzzy_max_expansions element in query_string query is not allowed.
Max_expansions element in a fuzzy query is not allowed.
When specifying the URL query param ’scroll’ it can not be greater than 6m.
To retrieve the accurate number of hits of your query
add to your request: "track_total_hits":true.
If you are running OpenSearch API requests with scripts
note that there is a 160 requests limit per 30 seconds.

When using the scroll API _search/scroll

Supported top-level elements of the scroll API: size, scroll, scroll_ids.
Scroll element cannot be greater than 6m. This value is the maximum amount of time the _scroll_id will be valid after the last scroll API request. If for example a script is used for the second request, over and over again, then this time limit will have no effect. The countdown starts when the scroll API requests stop. When the time expires, let's say it was set originally to 6 minutes, then if an additional scroll API request is sent, it will fail, as the _scroll_id is no longer valid.
Size element cannot be greater than 12000.

Additional resources


OpenSearch	OpenSearch API documentation

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].

Hosted OpenSearch View

Next Hosted Grafana View