gRPC API queries

Overview

Using gRPC offers several advantages for developers building high-performance integrations. Because gRPC relies on Protocol Buffers (Protobuf) instead of JSON, requests and responses are transmitted in a compact binary format. This results in smaller payloads, faster serialization and deserialization, and lower CPU and memory overhead compared to traditional HTTP/JSON APIs. The result: reduced latency, higher throughput, and more efficient use of network resources.

Requirements

To make gRPC API calls, you’ll need a tool like grpcurl. You can install grpcurl or use any tool of your choice, but all of the examples below use grpcurl.

To use this API you need to create a personal or team API key. It’s recommended to use permission presets, as they are automatically updated with all relevant permissions. Alternatively, you can manually add individual permissions.

Any request that lacks proper authentication will be rejected. To authenticate, add an Authorization Bearer <cx_api_key> header to your API request. It contains a Bearer Token, which identifies a single user, bot user, or workspace-application relationship for authentication.

Select the https://api./api/v1/dataprime/query endpoint that corresponds to your Coralogix domain using the domain selector at the top of the page.

A typical gRPC API call is formatted like this: grpcurl -H "Authorization: <cx_api_key>" -d "<data_json_object>" <host_name> <grpc_method>

Request payload body

When making a POST request, structure your payload using the format shown below. The query field is required; all other fields are optional and can be included in the metadata object to control execution parameters such as time range, query tier, or default source.

The following example consists of a JSON structure that represents the request:

{
  "query": "source logs | limit 100",
  "metadata": {                                    // metadata object is optional
    "tier": "TIER_ARCHIVE",                        // Search target of the query. Options: TIER_ARCHIVE, TIER_FREQUENT_SEARCH. Default: TIER_FREQUENT_SEARCH
    "syntax": "QUERY_SYNTAX_DATAPRIME",            // Query syntax. Options: QUERY_SYNTAX_DATAPRIME, QUERY_SYNTAX_LUCENE, others (see Swagger). Default: QUERY_SYNTAX_DATAPRIME
    "startDate": "2023-05-29T11:20:00.00Z",        // ISO 8601 date-time string defining the beginning of the query timeframe
    "endDate": "2023-05-29T11:30:00.00Z",          // ISO 8601 date-time string defining the ending of the query timeframe
    "defaultSource": "logs",                       // Default source used when source is omitted in a query
    "limit": 100,                                  // int32; limits the number of returned results. Max: 50,000 (TIER_ARCHIVE) or 12,000 (TIER_FREQUENT_SEARCH). Default: 2000
    "strictFieldsValidation": false,               // true = fail on unknown fields. Default: false
    "nowDate": "2023-05-29T12:00:00.00Z"           // substitutes for now() in query. Default: current time
  },
  "executionProfile": {                            // optional; controls the query engine's accuracy/performance trade-off
    "preset": "EXECUTION_PROFILE_PRESET_ACCURACY"  // EXECUTION_PROFILE_PRESET_ACCURACY (default): fresh results from source. EXECUTION_PROFILE_PRESET_PERFORMANCE: use caches for faster, approximate results
  }
}

For the full request and response schema reference (including all metadata fields, enum values, and limits), see the DataPrime Query Service Swagger reference.

Direct API query

com.coralogixapis.dataprime.v1.DataprimeQueryService/Query

message Query {
    string query = 1;
}

{
  "query": "source logs | limit 10"
}

Make the API call

grpcurl -H "Authorization: <cx_api_key>" \
    -d '{ "query": "source logs | limit 10", "metadata": { "tier": "TIER_ARCHIVE", "syntax": "QUERY_SYNTAX_DATAPRIME" }}' \
    api.:443 \
    com.coralogixapis.dataprime.v1.DataprimeQueryService/Query

Response

gRPC responses are streamed as binary data and may arrive in multiple messages instead of a single response object, as in REST. For background queries, the server sends the queryId first to acknowledge receipt, followed by a separate message containing the actual result data. These parts are logically connected but delivered in sequence over the stream. In most tools, such as grpcurl, these streamed messages are printed as separate JSON blocks.

For the full response schema (including all result and error field shapes), see the QueryResponse reference.

{
  "queryId": {
    "queryId": "a1e02902-9ecf-490d-8b3f-ce9dba179e04"
  }
}
{
  "result": {
    "results": [
      {
        "metadata": [
          ...
        ],
        "labels": [
          ...
        ],
        "userData": ...
      },
      ...
    ]
  }
}
{
  "statistics": {
    "status": "COMPLETED",
    "e2eDurationMs": "260",
    "outputRowCount": "1",
    "storage": {
      "objectStore": {
        "bytesRead": "0",
        "crossRegionBytesRead": "0",
        "limits": {
          "scan": { "reached": false, "limit": "50000000000", "measuredValue": "454108" },
          "shuffleSize": { "reached": false, "limit": "1000000000", "measuredValue": "704" },
          "filesRead": { "reached": false, "measuredValue": "0" },
          "column": { "reached": false }
        }
      }
    }
  }
}

Every direct query response ends with a statistics object describing execution status, duration, row count, bytes read, and per-resource limit signals. Integer values are returned as JSON strings to preserve 64-bit precision. For the full field reference, see Query statistics on the HTTP API page. The same statistics model is returned by GetBackgroundQueryStatus once a background query reaches a terminated state.

Background queries gRPC API overview

The Background queries gRPC API enables you to run high-latency, long-running queries asynchronously using DataPrime or Lucene syntax from scripts or the CLI. Designed for recurring, extensive analytical tasks—such as monthly or quarterly reports—this API allows you to offload complex queries to run in the background, freeing you to continue real-time monitoring within the Coralogix platform.

Use it to:

• Seamlessly query archived logs and spans, regardless of time range.
• Leverage higher scan, file, and shuffle limits for deeper data access.
• Take advantage of extended execution times for heavy analytical workloads.

The API supports the following background gRPCs:

• SubmitBackgroundQuery
• GetBackgroundQueryStatus
• CancelBackgroundQuery
• GetBackgroundQueryData

SubmitBackgroundQuery

com.coralogixapis.dataprime.v1.DataprimeQueryService/SubmitBackgroundQuery

message SubmitBackgroundQueryRequest {
  google.protobuf.StringValue query = 1;
  com.coralogixapis.dataprime.v1.QuerySyntax syntax = 2;
  // Start of query time range (inclusive)
  optional google.protobuf.Timestamp start_date = 3;
  // End of query time range (exclusive)
  optional google.protobuf.Timestamp end_date = 4;
  // Used by functions like now() in DataPrime
  optional google.protobuf.Timestamp now_date = 5;
  // Where to write results. When omitted, writes to an engine-owned temporary
  // dataset (system/engine.resultsets.bg_queries.{unique_id}) with 30-day retention.
  optional ResultsDestination results_destination = 6;
  // Fully qualified name (FQN) of the default source when omitted in the query. Defaults to "default/logs".
  optional string default_source = 7;
  // Optional. Starts with up to two underscores followed by a letter, then letters,
  // numbers, underscores, or dots; no consecutive or trailing dots; 2-255 characters.
  // Defaults to an auto-generated name.
  optional string name = 8;
  // Optional description, up to 2048 characters. Defaults to empty string.
  optional string description = 9;
}

{
  "query": "limit 3", 
  "syntax":"QUERY_SYNTAX_DATAPRIME", 
  "startDate": "2024-01-20T00:00:00Z", 
  "endDate": "2024-01-20T01:00:00Z", 
  "nowDate":"2024-01-20T02:00:00Z"
  }

Make the API call

 grpcurl -H "Authorization: <cx_api_key>" \
    -d '{ "query": "limit 3", "syntax":"QUERY_SYNTAX_DATAPRIME", "startDate": "2024-01-20T00:00:00Z", "endDate": "2024-01-20T01:00:00Z", "nowDate":"2024-01-20T02:00:00Z"}' \
    api.:443 \
    com.coralogixapis.dataprime.v1.DataprimeQueryService/SubmitBackgroundQuery

Response

The response object contains a queryId. This ID is used to do follow up operations on the background query.

{
  "queryId": "1712b991-b9be-455f-944e-308f6aa6bbbe"
}

Writing results to a dataset

To write results to a persistent, user-defined dataset instead of the temporary default, set resultsDestination. The target dataset must already exist under the default dataspace (see Background Queries for the in-app equivalent and Dataspace Management to create one):

grpcurl -H "Authorization: <cx_api_key>" \
    -d '{
      "query": "source logs | limit 1000",
      "syntax": "QUERY_SYNTAX_DATAPRIME",
      "startDate": "2024-01-20T00:00:00Z",
      "endDate": "2024-01-20T01:00:00Z",
      "resultsDestination": {
        "persistentDestination": {
          "datasets": [
            { "dataspace": "default", "dataset": "my_summary_dataset", "writeMode": "DATASET_WRITE_MODE_APPEND" }
          ],
          "partitioning": { "dateHourPartitioned": { "dataprimePath": "$m.timestamp" } }
        }
      }
    }' \
    api.:443 \
    com.coralogixapis.dataprime.v1.DataprimeQueryService/SubmitBackgroundQuery

The destination fields use the following message types.

ResultsDestination

Determines where the query writes its results. Set at most one of temporaryDestination or persistentDestination. When resultsDestination is omitted, results go to the temporary destination.

message ResultsDestination {
  oneof kind {
    // Results stored in the engine-owned system/engine.resultsets.bg_queries.{unique_id} dataset for 30 days.
    TemporaryDestination temporary_destination = 1;
    // Results written to a user-defined dataset in the caller's team.
    PersistentDestination persistent_destination = 2;
  }

  message TemporaryDestination {}

  message PersistentDestination {
    // v1 accepts at most one entry. Submitting more than one returns a BAD_REQUEST error.
    repeated DatasetDestination datasets = 1;
    // When omitted, uses the same partitioning as the query's results.
    optional Partitioning partitioning = 2;
  }
}

DatasetDestination

message DatasetDestination {
  // The dataset must already exist; this API does not create datasets, and it must be under the `default` dataspace.
  google.protobuf.StringValue dataspace = 1;
  google.protobuf.StringValue dataset = 2;
  DatasetWriteMode write_mode = 3;

  enum DatasetWriteMode {
    DATASET_WRITE_MODE_UNSPECIFIED = 0;
    DATASET_WRITE_MODE_APPEND = 1;
    DATASET_WRITE_MODE_OVERWRITE = 2;
  }
}

Partitioning

message Partitioning {
  oneof kind {
    Unpartitioned unpartitioned = 1;
    DateHourPartitioned date_hour_partitioned = 2;
  }

  // Partition results on values of timestamp type with date/hour granularity.
  message DateHourPartitioned {
    // DataPrime keypath of a timestamp to partition on, e.g. `$m.timestamp`.
    // The keypath must exist in the query's results and be of timestamp type.
    string dataprime_path = 1;
  }

  message Unpartitioned {}
}

GetBackgroundQueryStatus

com.coralogixapis.dataprime.v1.DataprimeQueryService/GetBackgroundQueryStatus

message GetBackgroundQueryStatusRequest {
  // Generated query ID that can be later used to obtain status and results
  string query_id = 1;
}

{ 
  "queryId": "412036c3-04be-431a-b1e2-9ebf971be6c6" 
}

Make the API call

grpcurl -H "Authorization: <cx_api_key>" \
    -d '{ "queryId": "412036c3-04be-431a-b1e2-9ebf971be6c6" }' \
    api.:443 \
    com.coralogixapis.dataprime.v1.DataprimeQueryService/GetBackgroundQueryStatus

Response

The response is a GetBackgroundQueryStatusResponse whose status oneof is one of running, terminated, or waitingForExecution. A terminated query carries its own success, error, or cancelled outcome:

message GetBackgroundQueryStatusResponse {
  // Status of the query
  oneof status {
    Running running = 1;
    Terminated terminated = 2;
    WaitingForExecution waiting_for_execution = 3;
  }

  // Returned when the query is waiting for execution.
  message WaitingForExecution {}

  // Returned when the query is running.
  message Running {
    // Time at which the query started running.
    google.protobuf.Timestamp running_since = 1;
  }

  // Returned when the query has terminated.
  message Terminated {
    google.protobuf.Timestamp running_since = 2;
    google.protobuf.Timestamp terminated_at = 3;
    oneof status {
      Success success = 4;
      Error error = 5;
      Cancelled cancelled = 6;
    }

    message Success {}
    message Cancelled {}

    message Error {
      oneof error {
        TimedOut timed_out = 1;
        Cancelled cancelled = 2;
        Failed failed = 3;
      }

      message TimedOut {}
      message Cancelled {}
      message Failed {
        google.protobuf.StringValue reason = 1;
      }
    }
  }
}

Example (terminated):

{
  "terminated": {
    "runningSince": "2025-01-21T09:35:01Z",
    "terminatedAt": "2025-01-21T09:35:01Z",
    "success": {}
  },
  "submittedAt": "2025-01-21T09:35:00Z",
  "metadata": [
    {
      "statistics": {
        "bytesScanned": "506070"
      }
    }
  ]
}

CancelBackgroundQuery

com.coralogixapis.dataprime.v1.DataprimeQueryService/CancelBackgroundQuery

message CancelBackgroundQueryRequest {
  // Generated query ID that can be later used to obtain status and results
  string query_id = 1;
}

{
  "queryId": "412036c3-04be-431a-b1e2-9ebf971be6c6"
}

Make the API call

grpcurl -H "Authorization: <cx_api_key>" \
    -d '{ "queryId": "412036c3-04be-431a-b1e2-9ebf971be6c6" }' \
    api.:443 \
    com.coralogixapis.dataprime.v1.DataprimeQueryService/CancelBackgroundQuery

Response

{}

GetBackgroundQueryData

com.coralogixapis.dataprime.v1.DataprimeQueryService/GetBackgroundQueryData

message GetBackgroundQueryDataRequest {
  // Generated query ID that can be later used to obtain status and results
  string query_id = 1;
}

{
  "queryId": "412036c3-04be-431a-b1e2-9ebf971be6c6"
}

Make the API call

grpcurl -H "Authorization: <cx_api_key>" \
    -d '{ "queryId": "412036c3-04be-431a-b1e2-9ebf971be6c6" }' \
    api.:443 \
    com.coralogixapis.dataprime.v1.DataprimeQueryService/GetBackgroundQueryData

Response

{
  "queryId": {
    "queryId": "a1e02902-9ecf-490d-8b3f-ce9dba179e04" // initial acknowledgment
  }
}
{
  "result": {  // returned later via streaming
    "results": [
      {
        "metadata": [
          ...
        ],
        "labels": [
          ...
        ],
        "userData": ...
      },
      ...
    ]
  }
}

Response handling

gRPC APIs use standard gRPC status codes instead of traditional HTTP response codes. When a request fails, the response will include one of the following codes, along with a descriptive error message.

Status code	Description
OK	The request was successful.
INVALID_ARGUMENT	The request was malformed — e.g., a required field was missing or misformatted.
UNAUTHENTICATED	The API key is missing or invalid.
PERMISSION_DENIED	The API key is valid but lacks the required permissions.
NOT_FOUND	The specified queryId does not exist or has expired.

A response with INVALID_ARGUMENT usually indicates that the request body doesn't follow the expected schema or contains invalid enum values or date formats.

If you receive UNAUTHENTICATED or PERMISSION_DENIED, check your API keys page or contact your workspace administrator to ensure you have the correct credentials and permissions.

Limitations

When a limit is breached, a warning message is displayed.

See the limitations page for a comprehensive list of direct query limitations and background query limitations.

Additional resources