# `engine.queries` ## Purpose The `engine.queries` dataset logs detailed information about the execution of queries within your environment. It captures both **semantic details** (e.g., query structure, labels, joins) and **execution-level statistics** (e.g., performance metrics, errors, resource usage). This dataset is invaluable for **query performance analysis**, helping teams investigate slow queries, diagnose failures, and track usage patterns. By understanding the nuances of query execution, teams can identify inefficiencies, optimize query performance, and ensure the reliability of their data operations. Some of the many use cases for `engine.queries` include: - [Finding the queries that take up the most resources](https://coralogix.com/docs/dataprime/cookbook/find_heavy_queries/index.md) - [Counting queries by which team made them](https://coralogix.com/docs/dataprime/cookbook/count_queries_by_team/index.md) - [Finding and grouping failed queries by failure type](https://coralogix.com/docs/dataprime/cookbook/failed_query_analysis/index.md) ## Schema description | Full JSON path | Field data type | Field data example | Description | | ---------------------------------------------------------------------------------------- | --------------- | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `clientInfo.userEmail` | String | `"alex@acme.io"` | The email address of the user who issued the query. | | `clientInfo.apiKeyName` | String | `"ci-runner-key"` | Name of the API key that issued the query. Empty when a user issued the query directly. | | `clientInfo.originatingTeamId` | String | `"42"` | Internal team ID that owns or initiated the query. | | `queryInfo.semanticLabels` | Object | `{...}` | Aggregated booleans describing structural/query features. | | `queryInfo.semanticLabels.containsFreeTextSearch` | Boolean | `true` | Whether the query includes full-text search elements. | | `queryInfo.semanticLabels.containsUnions` | Boolean | `false` | Whether SQL-style `UNION` clauses are present. | | `queryInfo.semanticLabels.containsAggregations` | Boolean | `true` | Whether aggregate functions (e.g., `SUM`, `COUNT`, `AVG`) are used. | | `queryInfo.semanticLabels.nonDefaultLimitRequested` | Boolean | `true` | Indicates a non-default result size was requested. | | `queryInfo.semanticLabels.containsWildTextSearch` | Boolean | `false` | Whether wildcard search terms (e.g., `*`, `%`) are used. | | `queryInfo.semanticLabels.containsGroupingSets` | Boolean | `false` | Whether grouping sets or similar constructs are used. | | `queryInfo.semanticLabels.containsJoins` | Boolean | `true` | Whether the query joins multiple datasets/tables. | | `queryInfo.semanticLabels.containsExtract` | Boolean | `true` | Whether extract operations (e.g., parsing time/strings) are used. | | `queryInfo.semanticLabels.containsWriteto` | Boolean | `false` | Whether the query uses a `writeto` clause to materialize results into a dataset. | | `queryInfo.semanticLabels.extractBeforeFilter` | Boolean | `false` | Whether the extract occurs before filtering in execution order. | | `queryInfo.querySyntax` | Enum | `"dataprime"` | Syntax format used by the query: `dataprime`, `lucene`, or `opensearch`. | | `queryInfo.interfaceType` | Enum | `"dataprime"` | Interface that the query was submitted against: `dataprime` or `opensearch`. | | `queryInfo.tier` | Enum | `"low"` | Query priority tier: `high`, `medium`, or `low`. | | `queryInfo.sources` | Array | `[{"fqDataset":"default/logs","teamId":"42","timeFrame":{...}}]` | Source datasets and configurations used in the query. | | `queryInfo.sources.fqDataset` | String | `"default/logs"` | Fully-qualified dataset name (`/`). | | `queryInfo.sources.teamId` | String | `"42"` | Team responsible for the dataset (optional). | | `queryInfo.sources.timeFrame` | Object | `{"start":1777545819954000000,"end":1777841999999000000,"durationMs":296180045}` | Selected time range for the source. | | `queryInfo.sources.timeFrame.start` | Number | `1777545819954000000` | Start of the source's time range, in epoch nanoseconds. | | `queryInfo.sources.timeFrame.end` | Number | `1777841999999000000` | End of the source's time range, in epoch nanoseconds. | | `queryInfo.sources.timeFrame.durationMs` | Number | `296180045` | Total timeframe duration in milliseconds. | | `queryInfo.sources.scopeExpression` | String | `"service='api' AND env='prod'"` | Expression that filters or scopes the source data. | | `queryInfo.queryOutcome` | Object | `{...}` | Final execution outcome details. | | `queryInfo.queryOutcome.errorMessage` | String | `"Syntax error near 'FROM'"` | Optional error message if the query failed. | | `queryInfo.queryOutcome.outputRowCount` | Number | `124` | Number of rows returned. | | `queryInfo.queryOutcome.status` | Enum | `"Completed"` | Final status: `Completed`, `Failed`, `Cancelled`, `TimedOut`, or `Incomplete`. | | `queryInfo.queryOutcome.storage` | Object | `{...}` | Storage and execution metadata: where the query ran, which storage locations it touched, what it produced, and resource statistics. | | `queryInfo.queryOutcome.storage.home` | Object | `{"cloud":"aws","aws":{"region":"eu-west-1"}}` | Cloud and region where the query was executed. | | `queryInfo.queryOutcome.storage.home.cloud` | Enum | `"aws"` | Cloud provider hosting the query engine. | | `queryInfo.queryOutcome.storage.home.aws.region` | String | `"eu-west-1"` | AWS region where the query was executed. Present when `cloud` is `aws`. | | `queryInfo.queryOutcome.storage.locations` | Array | `[{"type":"objectStore","locationOwner":"customer","objectStore":{...}}]` | Storage locations the query read from or wrote to during execution. Each item is one of two variants — `objectStore` (customer-owned) or `high` (provider-owned). | | `queryInfo.queryOutcome.storage.locations.type` | Enum | `"objectStore"` | Location variant: `objectStore` or `high`. | | `queryInfo.queryOutcome.storage.locations.locationOwner` | Enum | `"customer"` | `customer` for `objectStore` locations; `provider` for `high` locations. | | `queryInfo.queryOutcome.storage.locations.objectStore` | Object | `{...}` | Object-store metadata. Present when `type` is `objectStore`. | | `queryInfo.queryOutcome.storage.locations.objectStore.type` | Enum | `"aws_s3"` | Object-store type: `aws_s3`, `gcp_gs`, `azure_blobStorage`, or `ibm_cos`. | | `queryInfo.queryOutcome.storage.locations.objectStore.aws_s3.bucket` | String | `"customer-archive-bucket"` | S3 bucket name. Present when `objectStore.type` is `aws_s3`. | | `queryInfo.queryOutcome.storage.locations.objectStore.aws_s3.region` | String | `"eu-west-1"` | S3 bucket region. Present when `objectStore.type` is `aws_s3`. | | `queryInfo.queryOutcome.storage.locations.objectStore.stats` | Object | `{...}` | Object-store I/O stats for this location. | | `queryInfo.queryOutcome.storage.locations.objectStore.stats.bytesRead` | Number | `0` | Bytes read from this object-store location. | | `queryInfo.queryOutcome.storage.locations.objectStore.stats.getRequests` | Number | `0` | Number of `GET` requests issued to this object-store location. | | `queryInfo.queryOutcome.storage.locations.objectStore.stats.headRequests` | Number | `0` | Number of `HEAD` requests issued to this object-store location. | | `queryInfo.queryOutcome.storage.locations.high` | Object | `{...}` | Provider-owned storage metadata. Present when `type` is `high`. | | `queryInfo.queryOutcome.storage.locations.high.stats.bytesRead` | Number | `0` | Bytes read from provider-owned storage. | | `queryInfo.queryOutcome.storage.outputs` | Array | `[{"type":"writeto","writeto":{...}}]` | Outputs produced by the query (e.g., results materialized via `writeto`). | | `queryInfo.queryOutcome.storage.outputs.type` | Enum | `"writeto"` | Output type. | | `queryInfo.queryOutcome.storage.outputs.writeto` | Object | `{...}` | `writeto` output details. Present when `type` is `writeto`. | | `queryInfo.queryOutcome.storage.outputs.writeto.dataset.stats.totalBytesWritten` | Number | `87893` | Total compressed bytes written across all targets. | | `queryInfo.queryOutcome.storage.outputs.writeto.dataset.stats.totalUncompressedBytes` | Number | `24469` | Total uncompressed bytes written across all targets. | | `queryInfo.queryOutcome.storage.outputs.writeto.dataset.targets` | Array | `[{...}]` | Per-target `writeto` results. | | `queryInfo.queryOutcome.storage.outputs.writeto.dataset.targets.dataset` | String | `"materialized_events"` | Target dataset name. | | `queryInfo.queryOutcome.storage.outputs.writeto.dataset.targets.dataspace` | String | `"default"` | Target dataspace. | | `queryInfo.queryOutcome.storage.outputs.writeto.dataset.targets.writeMode` | Enum | `"append"` | Write mode used: `append` or `overwrite`. | | `queryInfo.queryOutcome.storage.outputs.writeto.dataset.targets.stats.bytesWritten` | Number | `87893` | Compressed bytes written to this target. | | `queryInfo.queryOutcome.storage.outputs.writeto.dataset.targets.stats.uncompressedBytes` | Number | `24469` | Uncompressed bytes written to this target. | | `queryInfo.queryOutcome.storage.stats` | Object | `{...}` | Execution-level resource statistics for the query. | | `queryInfo.queryOutcome.storage.stats.bytesRead` | Number | `7271615` | Bytes read from customer-exposed storage (customer bucket or provider-owned storage). Excludes cache and staging bucket reads. | | `queryInfo.queryOutcome.storage.stats.crossRegionBytesRead` | Number | `0` | Bytes read from a different region than the DataPrime Query Engine (DQE) cluster's region. | | `queryInfo.queryOutcome.storage.stats.limits` | Object | `{...}` | Per-resource limits. Each entry includes `reached` (whether the limit was hit) and, where applicable, `limit` (the threshold) and `measuredValue` (the measured amount). Threshold values vary by tier and account configuration; the `limit` examples in the rows below are illustrative only. | | `queryInfo.queryOutcome.storage.stats.limits.scan.reached` | Boolean | `false` | `true` if the scan limit was reached. | | `queryInfo.queryOutcome.storage.stats.limits.scan.limit` | Number | `1073741824` | Scan-byte threshold. | | `queryInfo.queryOutcome.storage.stats.limits.scan.measuredValue` | Number | `14522` | Bytes the engine scanned (may include data from in-memory cache, not just storage). | | `queryInfo.queryOutcome.storage.stats.limits.shuffleSize.reached` | Boolean | `false` | `true` if the shuffle size limit was reached. | | `queryInfo.queryOutcome.storage.stats.limits.shuffleSize.limit` | Number | `1073741824` | Shuffle-byte threshold. | | `queryInfo.queryOutcome.storage.stats.limits.shuffleSize.measuredValue` | Number | `83456` | Bytes shuffled by the query. | | `queryInfo.queryOutcome.storage.stats.limits.filesRead.reached` | Boolean | `false` | `true` if the files-read limit was reached. | | `queryInfo.queryOutcome.storage.stats.limits.filesRead.limit` | Number | `10000` | Files-read threshold. | | `queryInfo.queryOutcome.storage.stats.limits.filesRead.measuredValue` | Number | `0` | Number of files the query read. | | `queryInfo.queryOutcome.storage.stats.limits.aggBuckets.reached` | Boolean | `false` | `true` if the aggregation-buckets limit was reached. | | `queryInfo.queryOutcome.storage.stats.limits.aggBuckets.limit` | Number | `10000` | Aggregation-buckets threshold. | | `queryInfo.queryOutcome.storage.stats.limits.aggBuckets.measuredValue` | Number | `0` | Number of aggregation buckets used by the query. | | `queryInfo.queryOutcome.storage.stats.limits.column.reached` | Boolean | `false` | `true` if the column limit was reached. The column count itself is not emitted because it is computed at ingest, not at query time. | | `queryInfo.queryOutcome.storage.stats.limits.scrollTimeout.reached` | Boolean | `false` | `true` if the scroll operation timed out. | | `queryInfo.queryOutcome.failureType` | Enum | `"bad request"` | Failure reason (if applicable). | | `queryInfo.queryOutcome.failureClass` | Enum | `"clientError"` | Error class: `clientError` or `serverError`. | | `queryInfo.queryOutcome.e2eDurationMs` | Number | `842` | End-to-end execution time (ms). | | `queryInfo.queryId` | String | `"q-2025-09-04-abc123"` | Unique identifier for the query execution. | | `queryInfo.queryBlueprints` | Object | `{...}` | Normalized representations of query components. | | `queryInfo.queryBlueprints.queryTextSearchFilters` | String | `"text:\"payment failed\""` | Representation of text-based filters. | | `queryInfo.queryBlueprints.queryNoLiterals` | String | `"SELECT * FROM logs WHERE status = ?"` | Query string with literals removed. | | `queryInfo.queryBlueprints.queryLabelFilters` | String | `"service=api, env=prod"` | Normalized label-based filters. | | `queryInfo.defaultTimeFrame` | Object | `{"start":1777801523000000000,"end":1777805123000000000,"durationMs":3600000}` | Default time range if none is specified. | | `queryInfo.defaultTimeFrame.start` | Number | `1777801523000000000` | Default start time, in epoch nanoseconds. | | `queryInfo.defaultTimeFrame.end` | Number | `1777805123000000000` | Default end time, in epoch nanoseconds. | | `queryInfo.defaultTimeFrame.durationMs` | Number | `3600000` | Default time range duration in milliseconds. | | `queryInfo.queryText` | String | \`"source default/logs \\ | limit 10"\` | ## `engine.queries` schema { `clientInfo` `userEmail` type: `string`\ The email address of the user who issued the query. `apiKeyName` type: `string`\ Name of the API key that issued the query. Empty when a user issued the query directly. `originatingTeamId` type: `string`\ The internal team ID that owns or initiated the query. } { `queryInfo` { `semanticLabels` `containsFreeTextSearch` type: `boolean`\ Whether the query includes full-text search elements. `containsUnions` type: `boolean`\ Whether the query includes SQL-style `UNION` clauses. `containsAggregations` type: `boolean`\ Whether the query uses aggregation functions like `SUM`, `COUNT`, `AVG`. `nonDefaultLimitRequested` type: `boolean`\ Indicates if the query requests a result size different from the default. `containsWildTextSearch` type: `boolean`\ Whether the query includes wildcard search terms (e.g., `*`, `%`). `containsGroupingSets` type: `boolean`\ Whether grouping sets or similar constructs are used in the query. `containsJoins` type: `boolean`\ Whether the query involves joining multiple datasets or tables. `containsExtract` type: `boolean`\ Whether the query includes extract operations (e.g., parsing time or string fields). `containsWriteto` type: `boolean`\ Whether the query uses a `writeto` clause to materialize results into a dataset. `extractBeforeFilter` type: `boolean`\ Whether the extract operation occurs before filtering in execution order. } `querySyntax` Enum: `dataprime`, `lucene`, `opensearch`\ Syntax format used by the query. `interfaceType` Enum: `dataprime`, `opensearch`\ Interface that the query was submitted against. `tier` Enum: `high`, `medium`, `low`\ Query priority tier. { `sources` type: `array`\ A list of source datasets and their configurations used in the query. { items `fqDataset` type: `string`\ The fully qualified name of the dataset (`/`). `teamId` type: `string`\ The team responsible for the dataset (optional). { `timeFrame` `start` type: `number`\ Start of the source's time range, in epoch nanoseconds. `end` type: `number`\ End of the source's time range, in epoch nanoseconds. `durationMs` type: `number`\ Total duration of the selected timeframe in milliseconds. } `scopeExpression` type: `string`\ Expression that filters or scopes the source data (e.g., labels or conditions). } } { `queryOutcome` `errorMessage` type: `string`\ Optional error message if the query failed. `outputRowCount` type: `number`\ Number of rows returned by the query. `status` Enum: `Completed`, `Failed`, `Cancelled`, `TimedOut`, `Incomplete`\ The final status of the query execution. { `storage` { `home` `cloud` Enum: `aws`\ Cloud provider hosting the query engine. { `aws` `region` type: `string`\ AWS region where the query was executed. Present when `cloud` is `aws`. } } { `locations` type: `array`\ Storage locations the query read from or wrote to. Each item is one of two variants — `objectStore` (customer-owned) or `high` (provider-owned). { items `type` Enum: `objectStore`, `high`\ Location variant. `locationOwner` Enum: `customer`, `provider`\ `customer` for `objectStore` locations; `provider` for `high` locations. { `objectStore` Present when `type` is `objectStore`. `type` Enum: `aws_s3`, `gcp_gs`, `azure_blobStorage`, `ibm_cos`\ Object-store type. { `aws_s3` Present when `objectStore.type` is `aws_s3`. `bucket` type: `string`\ S3 bucket name. `region` type: `string`\ S3 bucket region. } { `stats` `bytesRead` type: `number`\ Bytes read from this object-store location. `getRequests` type: `number`\ Number of `GET` requests issued to this object-store location. `headRequests` type: `number`\ Number of `HEAD` requests issued to this object-store location. } } { `high` Present when `type` is `high`. { `stats` `bytesRead` type: `number`\ Bytes read from provider-owned storage. } } } } { `outputs` type: `array`\ Outputs produced by the query (e.g., results materialized via `writeto`). { items `type` Enum: `writeto`\ Output type. { `writeto` { `dataset` { `stats` `totalBytesWritten` type: `number`\ Total compressed bytes written across all targets. `totalUncompressedBytes` type: `number`\ Total uncompressed bytes written across all targets. } { `targets` type: `array`\ Per-target `writeto` results. { items `dataset` type: `string`\ Target dataset name. `dataspace` type: `string`\ Target dataspace. `writeMode` Enum: `append`, `overwrite`\ Write mode used for this target. { `stats` `bytesWritten` type: `number`\ Compressed bytes written to this target. `uncompressedBytes` type: `number`\ Uncompressed bytes written to this target. } } } } } } } { `stats` `bytesRead` type: `number`\ Bytes read from customer-exposed storage (customer bucket or provider-owned storage). Excludes cache and staging bucket reads. `crossRegionBytesRead` type: `number`\ Bytes read from a different region than the DataPrime Query Engine (DQE) cluster's region. { `limits` Per-resource limit signals. Each entry includes `reached` (whether the limit was hit) and, where applicable, `limit` (the threshold) and `measuredValue` (the measured amount). Threshold values vary by tier and account configuration. { `scan` `reached` type: `boolean`\ `true` if the scan limit was reached. `limit` type: `number`\ Scan-byte threshold. `measuredValue` type: `number`\ Bytes the engine scanned (may include in-memory cache, not just storage). } { `shuffleSize` `reached` type: `boolean`\ `true` if the shuffle size limit was reached. `limit` type: `number`\ Shuffle-byte threshold. `measuredValue` type: `number`\ Bytes shuffled by the query. } { `filesRead` `reached` type: `boolean`\ `true` if the files-read limit was reached. `limit` type: `number`\ Files-read threshold. `measuredValue` type: `number`\ Number of files the query read. } { `aggBuckets` `reached` type: `boolean`\ `true` if the aggregation-buckets limit was reached. `limit` type: `number`\ Aggregation-buckets threshold. `measuredValue` type: `number`\ Number of aggregation buckets used. } { `column` `reached` type: `boolean`\ `true` if the column limit was reached. The column count itself is not emitted because it is computed at ingest, not at query time. } { `scrollTimeout` `reached` type: `boolean`\ `true` if the scroll operation timed out. } } } } `failureType` Enum: `bad request`, `rate limit reached`, `business timeout`, `not found`, `permission denied`, `internal`, `resource exhausted`, `internal death`, `query failed`, `query timed out`\ The reason the query failed, if applicable. `failureClass` Enum: `clientError`, `serverError`\ Classification of the error as client-side or server-side. `e2eDurationMs` type: `number`\ Total execution time from request to response in milliseconds. } `queryId` type: `string`\ A unique identifier for the query execution. { `queryBlueprints` `queryTextSearchFilters` type: `string`\ A representation of the query's text-based filters. `queryNoLiterals` type: `string`\ The query string with literals removed for comparison/normalization. `queryLabelFilters` type: `string`\ A normalized form of the label-based filters in the query. } { `defaultTimeFrame` `start` type: `number`\ Default time range start, in epoch nanoseconds. `end` type: `number`\ Default time range end, in epoch nanoseconds. `durationMs` type: `number`\ Total default time range duration in milliseconds. } `queryText` type: `string`\ The original raw query as issued by the user. } ## Next steps Track field-level schema evolution over time with [`engine.schema_fields`](https://coralogix.com/docs/user-guides/data-layer/system_dataspace/engine-schema_fields/index.md).