Elastic API

Coralogix provides an Elastic API that allows you to query your hosted Elasticsearch instances securely and with ease. 

How to query your Coralogix Elastic API:

In order to use the Elasticsearch API it is necessary to:

1. Add a Coralogix token with each HTTP request. This key is found under: Data Flow –> API Keys –> Logs Query Key.
   
2. Use the Elasticsearch API endpoint for your Team per the following table:

3. The Index is a variable in the in the Endpoint. You can put there for example the name of  the log2metrics index  (*:*_log_metrics*) if you are querying Log2metrics index, or (*) if you are querying the regular index.

The API request should contain the following:

• Headers:
  ‘token:Logs Query Key’
  “Content-type: application/json”

• URL: https://<Elasticsearch-api>/<Index>/_search
  (For example: https://esapi.coralogix.us:9443/*/_search/)
  

Examples:

curl -H 'token:Logs Query Key' -H "Content-type: application/json" -d '{
    "query": {
        "bool": {
            "must":
            [
                {
                    "term": {
                        "coralogix.metadata.applicationName": "PROD"
                    }
                },
                {
                    "range": {
                        "coralogix.timestamp": {
                            "gte": "now-15m",
                            "lt": "now"
                        }
                    }
                }
            ]
        }
    },
    "aggs": {
        "severities": {
            "terms": {
                "field": "coralogix.metadata.severity"
            }
        }
    }
}'

curl -H 'token:Logs Query Key' -H "Content-type: application/json" -d '{
    "query": {
        "bool": {
            "must":
            [
        {
            "match": {
                "text": "created"
            }
        },
                {
            "range": {
                "coralogix.timestamp": {
                    "gte": "now-15m", 
                    "lt": "now"
                    }
                }
             }
            ]
        }
    }
}'

curl -H 'token:Logs Query Key' -H "Content-type: application/json" -d '{
    "query": {
        "bool": {
            "filter":[{
                "query_string": {
                    "query": "YOUR QUERY"
                }
             },
                {
            "range": {
                "coralogix.timestamp": {
                    "gte": "2019-10-23T14:00:00",
                    "time_zone": "+03:00"
                }
            }
                }
            ]
        }
    }
}'

How to use the Elasticsearch scroll API:

The initial scroll API request should contain the following:

• Headers:
  ‘token:Logs Query Key’
  “Content-type: application/json”

• URL: https://<Elasticsearch-api>/<Index>/_search?scroll=5m
  (For example: https://esapi.coralogix.us:9443/*/_search?scroll=5m)

curl -H 'token:Logs Query Key' -H "Content-type: application/json" -d '{
	"size": 1000,
	"query": {
		"bool": {
			"filter": [{
					"query_string": {
						"query": "YOUR QUERY"
					}
				},
				{
					"range": {
						"coralogix.timestamp": {
							"gte": "now-24h",
							"lt": "now"
						}
					}
				}
			]
		}
	}
}'

You will receive the first batch of logs along with a new field in the root of the response named _scroll_id.

That same scroll_id should be used in subsequent requests to create the pagination, and obtain the next log batch. You should repeat the second request until all logs are retrieved.

Second and subsequent scroll API requests, should contain the following:

• Headers:
  ‘token:Logs Query Key’
  “Content-type: application/json”

• URL: https://<Elasticsearch-api>/_search/scroll
  (For example: https://esapi.coralogix.us:9443/_search/scroll)
  
  Please note that an index is not specified in the endpoint for the second, and subsequent requests.
  

Example #1:

curl -H 'token:Logs Query Key' -H "Content-type: application/json" -d '{

    "scroll": "5m",
    "scroll_id": "YOUR_SCROLL_ID"

}'

Example #2:

curl --location --request POST 'https://esapi.coralogix.us:9443/_search/scroll/' \
--header 'token: Logs Query Key' \
--header 'Content-Type: application/json' \
--data-raw '{
    "scroll": "5m",
    "scroll_id": "YOUR_SCROLL_ID"

}'

The last log page retrieved will show and empty array at the bottom of the page:

"hits": []

This indicates that there are no more log pages to retrieve.

Limitations

The Coralogix Elastic API provides the capabilities of the Elasticsearch API with the following limitations:

• Supports only POST requests.
• Supported top-level elements of the Search API: query, from, size, 
  sort, _source, post_filter, aggs, aggregations.
• The sum of the top-level elements ‘from’ and ‘size’ cannot be
  greater than 12000.
• allow_leading_wildcard element in query_string query is not allowed.
• Wildcard queries can’t start with ‘*’ or ‘?’
• RegEx queries can’t start with ‘.*’ or ‘.?’
• max_determinized_states element inside regex queries is not allowed.
• Size element for bucket aggregations cannot be greater than 1200.
• The bucket aggregation of the type significant_terms is not allowed.
• Nesting of the following bucket aggregations 3 or more times is not allowed: date_histogram, geohash_grid, histogram, 
  ip_ranges, and terms.
• fuzzy_max_expansions element in query_string query is not allowed.
• Max_expansions element in a fuzzy query is not allowed.
• When specifying the URL query param ’scroll’ it can not be greater than 6m.
• To retrieve the accurate number of hits of your query
  add to your request: “track_total_hits”:true.
• If you are running Elasticsearch API requests with scripts
  note that there is a 160 requests limit per 30 seconds.

When using the scroll API _search/scroll

• Supported top-level elements of the scroll API: size, scroll, scroll_ids.
• Scroll element cannot be greater than 6m
  (Please note that this value is the maximum amount of time the _scroll_id will be valid after the last scroll API request. If for example a script is used for the second request, over and over again, then this time limit will have no effect. The countdown starts when the scroll API requests stop. When the time expires, let’s say it was set originally to 6 minutes, then if an additional scroll API request is sent, it will fail, as the _scroll_id is no longer valid).
• Size element cannot be greater than 12000.

Elastic API query tutorials

1) Search API tutorial

2) Aggregations API tutorial