Background Queries API

The Background Queries API enables you to run high-latency, long-running queries asynchronously using DataPrime or Lucene syntax from scripts or CLI. Ideal for recurring, extensive analytical tasks—such as monthly or quarterly reports—this feature operates in the background, allowing you to continue active monitoring within the Coralogix platform.

This documentation outlines the API using a gRPC-style method. Documentation for the HTTP-style method is forthcoming.

Use it to:

Seamlessly query archived logs and spans, regardless of daily quota or time frame.
Leverage larger scan, file, and shuffle limits for comprehensive data retrieval.
Benefit from extended execution times for running complex queries.

How to send gRPC requests

STEP 1. Install grpcurl.

STEP 2. To use this API you need to create a personal or team API key. It’s recommended to use permission presets, as they are automatically updated with all relevant permissions. Alternatively, you can manually add individual permissions.

Preset	Action	Description
DataQuerying	`LEGACY-ARCHIVE-QUERIES:EXECUTE`	Query data from the archive

STEP 3. Run grpcurl with this template:

grpcurl -H "Authorization: <cx_api_key>" -d "<data_json_object>" <host_name> <grpc_method>

You can find the host_name here, depending on your Coralogix domain and region.

Supported API calls

The API supports the following gRPCs:

SubmitBackgroundQuery
GetBackgroundQueryStatus
GetBackgroundQueryData
CancelBackgroundQueryDataResponse

`com.coralogixapis.dataprime.v1.DataprimeQueryService/SubmitBackgroundQuery`

Request

message SubmitBackgroundQueryRequest {
  google.protobuf.StringValue query = 1;
  com.coralogixapis.dataprime.v1.QuerySyntax syntax = 2;
  // Start of query time range (inclusive)
  optional google.protobuf.Timestamp start_date = 3;
  // End of query time range (exclusive)
  optional google.protobuf.Timestamp end_date = 4;
  // Used by functions like now() in DataPrime
  optional google.protobuf.Timestamp now_date = 5;
}

Example:

grpcurl -H "Authorization: <cx_api_key>" \
    -d '{ "query": "limit 1", "syntax": "QUERY_SYNTAX_DATAPRIME", "start_date": "2024-01-20T00:00:00Z", "end_date": "2024-01-20T01:00:00Z", "now_date": "2024-01-20T02:00:00Z" }' \
    ng-api-grpc.<coralogix_url>:443 \
    com.coralogixapis.dataprime.v1.DataprimeQueryService/SubmitBackgroundQuery

Response

message SubmitBackgroundQueryResponse {
  // Generated query ID that can be later used to obtain status and results
  string query_id = 1;
  // All warnings that can appear during query submission
  repeated com.coralogixapis.dataprime.v1.DataprimeWarning warnings = 2;
}

Example:

{
  "queryId": "412036c3-04be-431a-b1e2-9ebf971be6c6"
}

`com.coralogixapis.dataprime.v1.DataprimeQueryService/GetBackgroundQueryStatus`

Request

message GetBackgroundQueryStatusRequest {
  // Generated query ID that can be later used to obtain status and results
  string query_id = 1;
}

Example:

grpcurl -H "Authorization: <cx_api_key>" \
    -d '{ "query_id": "412036c3-04be-431a-b1e2-9ebf971be6c6" }' \
    ng-api-grpc.<coralogix_url>:443 \
    com.coralogixapis.dataprime.v1.DataprimeQueryService/GetBackgroundQueryStatus

Response

message GetBackgroundQueryStatusResponse {
  // Status of the query
  oneof status {
    // Returned when the query is running
    Running running = 1;
    // Returned when the query 
    Terminated terminated = 2;
    // Returned when the query is waiting for execution
    WaitingForExecution waiting_for_execution = 3;
  }

Example:

{
  "terminated": {
    "runningSince": "2025-01-21T09:35:01Z",
    "terminatedAt": "2025-01-21T09:35:01Z",
    "success": {}
  },
  "submittedAt": "2025-01-21T09:35:00Z",
  "metadata": [
    {
      "statistics": {
        "bytesScanned": "506070"
      }
    }
  ]
}

`com.coralogixapis.dataprime.v1.DataprimeQueryService/GetBackgroundQueryData`

Request

message GetBackgroundQueryDataRequest {
  // Generated query ID that can be later used to obtain status and results
  string query_id = 1;
}

Example:

grpcurl -H "Authorization: <cx_api_key>" \
    -d '{ "query_id": "412036c3-04be-431a-b1e2-9ebf971be6c6" }' \
    ng-api-grpc.<coralogix_url>:443 \
    com.coralogixapis.dataprime.v1.DataprimeQueryService/GetBackgroundQueryData

Response

// Wrapper for future extensibility
message GetBackgroundQueryDataResponse {
  oneof message {
    GetDataResponse response = 1;
  }
}
// Wrapper for future extensibility
message GetDataResponse {
  com.coralogixapis.dataprime.v1.DataprimeResult results = 1;
}

Example:

{
  "response": {
    "results": {
      "results": [
        {
          "metadata": [...],
          "labels": [...],
          "userData": "..."
        },
        ...
      ]
    }
  }
}
{
  ...
}
...

`com.coralogixapis.dataprime.v1.DataprimeQueryService/CancelBackgroundQuery`

Request

message CancelBackgroundQueryRequest {
// Generated query ID that can be later used to obtain status and results
string query_id = 1;
}

Example:

grpcurl -H "Authorization: <cx_api_key>" \
    -d '{ "query_id": "412036c3-04be-431a-b1e2-9ebf971be6c6" }' \
    ng-api-grpc.<coralogix_url>:443 \
    com.coralogixapis.dataprime.v1.DataprimeQueryService/CancelBackgroundQuery

Response

// Message to cancel Background Query
message CancelBackgroundQueryResponse {
}

Example:

{}

Message types

WaitingForExecution

    // Message when query is waiting for execution
    message WaitingForExecution {
  }

Running

  // Message when query is running
  message Running {
    // Time at which the query started running
    google.protobuf.Timestamp running_since = 1;
  }

Terminated

message Terminated {
    // Time at which query started running
    google.protobuf.Timestamp running_since = 2;
    // Time at which the query was terminated
    google.protobuf.Timestamp terminated_at = 3;
    // Status of the query
    oneof status {
      Success success = 4;
      Error error = 5;
      Cancelled cancelled = 6;
    }

    message Success {
    }
    message Cancelled {
    }

    message Error {
      oneof error {
        TimedOut timed_out = 1;
        Cancelled cancelled = 2;
        Failed failed = 3;
      }

      message TimedOut {
      }

      message Cancelled {
      }

      message Failed {
        google.protobuf.StringValue reason = 1;
      }
    }
  }
}

Response handling

The APIs return standard responses using gRPC status codes. The supported status codes are as follows:

gRPC Code	Description	Method
`OK`	Success	All Methods
`INTERNAL`	Internal Server Error	All Methods
`UNAUTHENTICATED`	Unauthorized	All Methods
`PERMISSION_DENIED`	Forbidden	All Methods
`NOT_FOUND`	Metric or Rule ID not found	Allow Method
`INVALID_ARGUMENT`	Invalid Rule Expression or ID Format	Block/Allow Method
`ALREADY_EXISTS`	Metric Already Blocked	Block Method
`FAILED_PRECONDITION`	Bad Request	Block Method

Limitations

Background Query APIs are designed for extended operations and have the following limitations:

Execution time: Queries are limited to a maximum execution time of 30 minutes, compared to 5 minutes in Explore.
Data Latency: Results may take longer to process due to the extended time ranges and larger data scans supported by the API.

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email to [email protected].

Direct Archive Query HTTP API