engine.schema_fields
Purpose
The engine.schema_fields
dataset captures metadata and historical snapshots of dataset schemas within Coralogix's system
dataspace. This dataset provides a detailed record of a dataset's structural evolution over time, allowing users to track changes to dataset fields, their types, and other critical attributes. It is designed to help teams understand how the structure of a dataset has evolved, monitor data schema changes, and ensure that datasets maintain consistency across different versions.
The schema metadata includes information such as the field types, partitioning schemes, and labels, along with additional contextual data, such as dataset names and snapshot timestamps. This dataset is crucial for debugging, auditing schema changes, and ensuring data governance across datasets.
Some practical use cases of this dataset include:
- Counting schema changes over time
- Monitoring dataset schema changes for compliance
- Audit schema field types and values by dataset
Schema Description
Full JSON path | Field data type | Field data example | description |
---|---|---|---|
dataprimePath | String | "$d.dataset" | Path used for Dataprime processing. |
dataset | String | "engine.schema_fields" | The dataset name. |
dataspace | String | "system" | Dataspace this dataset belongs to. |
distinctValueCount | Number | 1 | Number of distinct values observed for this field. |
examples | Array\ | ["rumEventsPoc"] | Example values seen in the data. |
flatPath | String | "$d.dataset" | Flattened path representation used for referencing contexts. |
labels | Object | { "category":"io", "className":"com.acme.Handler", "methodName":"handle", "thread":"pool-1-7" } | Optional labels associated with the field. |
labels.category | String | "io" | Label describing a category for the field. |
labels.className | String | "com.acme.Handler" | Fully qualified class name (if applicable). |
labels.methodName | String | "handle" | Method name (if applicable). |
labels.thread | String | "pool-1-7" | Thread identifier/name (if applicable). |
metadata | Object | { "entityType":"log", "pillar":"observability", "priorityClass":"Medium", "severity":"3" } | Additional metadata about the field. |
metadata.entityType | String | "log" | Logical entity type for this field. |
metadata.pillar | String | "observability" | Product/solution pillar associated with the field. |
metadata.priorityClass | String | "Medium" | Priority classification. |
metadata.severity | String | "3" | Severity level. |
partitioningScheme | String | "dt/hr" | Partitioning scheme of the dataset. |
pathParts | Array\ | ["$d","dataset"] | Path breakdown used for data reference resolution. |
snapshotDuration | String (ns) | "3600000000000" | Duration of the snapshot in nanoseconds (string-encoded). |
snapshotId | String (UUID) | "043c5e02-0924-45f0-83dd-2aa5659e323e" | Unique identifier of the dataset snapshot. |
snapshotStartTime | Number (ns since epoch) | 1750611600000000000 | Start time of the snapshot in epoch nanoseconds. |
type | String | "string" | Field’s logical/type classification. |
engine.schema_fields
schema
Indicates the dataspace this dataset belongs to. Example: "system".
The dataset name. Example: "engine.schema_fields".
Flattened path representation used for referencing contexts. Example: "$d.dataset".
Path used for Dataprime processing. Example: "$d.dataset".
The first element of the path. Example: "$d".
The second element of the path. Example: "dataset".
Unique identifier of the dataset snapshot. Example: "043c5e02-0924-45f0-83dd-2aa5659e323e".
Start time of the snapshot in epoch nanoseconds. Example: 1750611600000000000.
Duration of the snapshot in epoch nanoseconds. Example: "3600000000000".
Partitioning scheme of the dataset. Example: "dt/hr".
Type of the field. Example: "string".
Example values of the dataset.
Example: ["rumEventsPoc"].
Number of distinct values. Example: 1.
Optional labels associated with the field.
High-cardinality labels, if any.
Priority classification. Example: "Medium".
Severity level. Example: "3".