Exposed metrics

Open in ChatGPTOpen in Claude

The following tables describe the metrics exported by OBI in both OpenTelemetry (OTEL) and Prometheus formats, for use within Coralogix observability workflows.

Application metrics

Family	Name (OTEL)	Name (Prometheus)	Type	Unit	Description
Application	http.client.request.duration	http_client_request_duration_seconds	Histogram	seconds	Time taken for outbound HTTP calls from client applications
Application	http.client.request.body.size	http_client_request_body_size_bytes	Histogram	bytes	Size of HTTP request payloads sent by clients
Application	http.client.response.body.size	http_client_response_body_size_bytes	Histogram	bytes	Size of HTTP responses received by clients
Application	http.server.request.duration	http_server_request_duration_seconds	Histogram	seconds	Time taken to process inbound HTTP requests on servers
Application	http.server.request.body.size	http_server_request_body_size_bytes	Histogram	bytes	Size of incoming HTTP request bodies received by servers
Application	http.server.response.body.size	http_server_response_body_size_bytes	Histogram	bytes	Size of outbound HTTP response bodies sent by servers
Application	rpc.client.duration	rpc_client_duration_seconds	Histogram	seconds	Duration of outbound gRPC service calls
Application	rpc.server.duration	rpc_server_duration_seconds	Histogram	seconds	Duration of inbound RPC service calls
Application	sql.client.duration	sql_client_duration_seconds	Histogram	seconds	Time taken for SQL client operations (Experimental)
Application	redis.client.duration	redis_client_duration_seconds	Histogram	seconds	Time taken for Redis client operations (Experimental)
Application	db.server.operation.duration	db_server_operation_duration_seconds	Histogram	seconds	Server-side database operation duration for Redis, Memcached, and SQL.
Application	messaging.publish.duration	messaging_publish_duration	Histogram	seconds	Time taken for message publication (e.g., Kafka) (Experimental)
Application	messaging.process.duration	messaging_process_duration	Histogram	seconds	Time taken for message processing (e.g., Kafka) (Experimental)
Application Process	process.cpu.time	process_cpu_time_seconds_total	Counter	seconds	Total CPU time consumed, broken down by state (system/user/wait)
Application Process	process.cpu.utilization	process_cpu_utilization_ratio	Gauge	ratio	CPU time usage change over time, normalized by CPU count
Application Process	process.memory.usage	process_memory_usage_bytes	UpDownCounter	bytes	Physical memory currently used
Application Process	process.memory.virtual	process_memory_virtual_bytes	UpDownCounter	bytes	Virtual memory committed
Application Process	process.disk.io	process_disk_io_bytes_total	Counter	bytes	Total disk I/O bytes transferred
Application Process	process.network.io	process_network_io_bytes_total	Counter	bytes	Total network I/O bytes transferred
Go runtime	go.memory.limit	go_memory_limit_bytes	UpDownCounter	bytes	Go runtime memory limit.
Go runtime	go.memory.gc.goal	go_memory_gc_goal_bytes	UpDownCounter	bytes	Committed GC heap goal. Omitted when the runtime offset and the fallback symbol are both unavailable.
Go runtime	go.memory.gc.cycles	go_memory_gc_cycles_total	Counter	count	Completed Go GC cycles.
Go runtime	go.memory.gc.pause.duration	go_memory_gc_pause_duration_seconds	Histogram	seconds	Cumulative stop-the-world GC pause duration. Requires Go 1.22+.
Go runtime	go.memory.used	go_memory_used_bytes	UpDownCounter	bytes	Memory in use, broken down by go.memory.type (stack or other). Requires Go 1.23+.
Go runtime	go.memory.allocated	go_memory_allocated_bytes_total	Counter	bytes	Cumulative allocated heap bytes. Requires Go 1.23+.
Go runtime	go.memory.allocations	go_memory_allocations_total	Counter	allocations	Cumulative heap allocation count. Requires Go 1.23+.
Go runtime	go.cpu.time	go_cpu_time_seconds_total	Counter	seconds	Cumulative CPU time, broken down by go.cpu.state (and go.cpu.detailed_state where applicable). Requires Go 1.23+.
Go runtime	go.goroutine.count	go_goroutine_count	UpDownCounter	goroutines	Current goroutine count. Omitted rather than reported partially when the runtime snapshot is incomplete.
Go runtime	go.processor.limit	go_processor_limit	UpDownCounter	processors	GOMAXPROCS value.
Go runtime	go.config.gogc	go_config_gogc_percent	UpDownCounter	percent	GOGC configuration value.
Go runtime	go.schedule.duration	go_schedule_duration_seconds	Histogram	seconds	Cumulative runnable-to-running goroutine latency. Requires Go 1.20+.
JVM	jvm.memory.used	jvm_memory_used_bytes	UpDownCounter	bytes	HotSpot JVM memory used, per memory pool. Sum by jvm.memory.type for heap and non-heap totals.
JVM	jvm.memory.committed	jvm_memory_committed_bytes	UpDownCounter	bytes	HotSpot JVM memory committed.
JVM	jvm.memory.limit	jvm_memory_limit_bytes	UpDownCounter	bytes	HotSpot JVM memory limit.
JVM	jvm.memory.used_after_last_gc	jvm_memory_used_after_last_gc_bytes	UpDownCounter	bytes	HotSpot JVM memory used after the last GC.
JVM	jvm.class.loaded	jvm_class_loaded_total	Counter	classes	Classes loaded since JVM start.
JVM	jvm.class.unloaded	jvm_class_unloaded_total	Counter	classes	Classes unloaded since JVM start.
JVM	jvm.class.count	jvm_class_count	UpDownCounter	classes	Classes currently loaded.
JVM	jvm.thread.count	jvm_thread_count	UpDownCounter	threads	Current executing thread count.
JVM	jvm.cpu.time	jvm_cpu_time_seconds_total	Counter	seconds	CPU time used by the JVM process.
JVM	jvm.cpu.count	jvm_cpu_count	UpDownCounter	processors	Processors available to the JVM.
JVM	jvm.cpu.recent_utilization	jvm_cpu_recent_utilization_ratio	Gauge	ratio	Recent CPU utilization of the JVM process.
Node.js runtime	nodejs.eventloop.utilization	nodejs_eventloop_utilization_ratio	Gauge	ratio	Event-loop utilization. Requires Node.js 14.10+.
Node.js runtime	nodejs.eventloop.time	nodejs_eventloop_time_seconds_total	Counter	seconds	Cumulative event-loop time, broken down by nodejs.eventloop.state (idle or active). Requires Node.js 14.10+.
Node.js runtime	nodejs.eventloop.delay.*	nodejs_eventloop_delay_*_seconds	Gauge	seconds	Event-loop delay statistics. min, max, mean, stddev, p50, p90, p99. Requires Node.js 16.14+.
Node.js runtime	v8js.gc.duration	v8js_gc_duration_seconds	Histogram	seconds	V8 garbage-collection pause duration.
Node.js runtime	v8js.memory.heap.used	v8js_memory_heap_used_bytes	UpDownCounter	bytes	V8 heap memory used.
Node.js runtime	v8js.memory.heap.limit	v8js_memory_heap_limit_bytes	UpDownCounter	bytes	V8 heap memory limit.
Node.js runtime	v8js.memory.heap.space.available_size	v8js_memory_heap_space_available_size_bytes	UpDownCounter	bytes	Available V8 heap-space memory.
Node.js runtime	v8js.memory.heap.space.physical_size	v8js_memory_heap_space_physical_size_bytes	UpDownCounter	bytes	Physical (committed) V8 heap-space memory.
Python runtime	cpython.gc.collections	cpython_gc_collections_total	Counter	collections	CPython garbage-collection runs.
Python runtime	cpython.gc.collected_objects	cpython_gc_collected_objects_total	Counter	objects	Objects reclaimed by the CPython garbage collector.
Python runtime	cpython.gc.uncollectable_objects	cpython_gc_uncollectable_objects_total	Counter	objects	Uncollectable objects found by the CPython garbage collector.
Network	obi.network.flow.bytes	obi_network_flow_bytes	Counter	bytes	Bytes transmitted between source and destination network endpoints
Network	obi.network.flow.packets	obi_network_flow_packets_total	Counter	count	Packets transmitted between source and destination network endpoints.
Network	obi.network.inter.zone.bytes	obi_network_inter_zone_bytes	Counter	bytes	Cross-zone network bytes within Kubernetes clusters (Experimental)
Network	obi.stat.tcp.io	obi_stat_tcp_io_bytes_total	Counter	bytes	TCP bytes transferred.
Network	obi.stat.tcp.retransmits	obi_stat_tcp_retransmits_total	Counter	count	TCP segment retransmissions.
Network	obi.stat.tcp.failed.connections	obi_stat_tcp_failed_connections_total	Counter	connections	TCP failed connection count, labeled by network.tcp.handshake.role (client or server) and failure reason. The reason label is disabled by default.

Enabling runtime metrics

Go, JVM, Node.js, and Python runtime metrics are all enabled through the shared application_runtime metrics feature; jvm_runtime_metrics.sampling_interval controls the JVM sampling rate. The Node.js event-loop and V8 metrics require the Node.js injector (nodejs.enabled, on by default). Each metric's introducing release is noted in the table above; see the OBI changelog for details.

Span and service-graph metrics

OBI also supports exporting Span metrics and Service Graph metrics, which can be toggled via feature configuration options.

The legacy application_span and application_span_sizes span-metric features are deprecated in favor of application_span_otel. The deprecated features still emit the Grafana-convention traces_spanmetrics_* metrics and keep working with a startup warning, while application_span_otel emits the OpenTelemetry-convention traces_span_metrics_* metrics. application_span_sizes has no direct replacement. The closest are http.server.request.body.size and http.server.response.body.size. Enabling a legacy feature together with application_span_otel is rejected, and the * (all) selector keeps only the OTel version.

Metric naming

Prometheus metric names are derived from their OTLP definitions rather than declared separately. On the OTLP export path, target_info, traces_target_info, traces_host_info, traces_span_metrics_calls_total, and traces_span_metrics_duration are emitted in dot notation as target.info, traces.target.info, traces.host.info, traces.span.metrics.calls, and traces.span.metrics.duration; the Prometheus scrape path keeps the underscore names. For heap totals, sum jvm.memory.used by jvm.memory.type (the OBI-specific obi.jvm.heap.used metric is not emitted). The OBI changelog lists the metric renames this introduced.

Span attributes

Database duration metrics carry the db.namespace, db.response.status_code, and server.port attributes, and messaging spans carry messaging.operation.name.

HTTP spans emit url.query by default when present, and client HTTP spans include the query string in url.full. Sensitive parameters are automatically redacted; extend the redaction list with OTEL_EBPF_SENSITIVE_QUERY_PARAMS_ADD. Java route harvesting is disabled by default, so the http.route attribute on Java spans falls back to the raw URL path.

Capture defaults

Database error response bodies are not included on SQL spans by default. Enable that capture explicitly if you need sql.client.duration spans to carry database error responses.

DNS calls are excluded from span metrics and service-graph metrics by default. SQL operations (MySQL, PostgreSQL, MariaDB) are reported as SERVER spans, so aggregations or filters that key on span kind for SQL traffic should account for that.

Attribute selection for OBI metrics

Metrics and attributes follow OTEL dot.notation but are converted to Prometheus underscore_notation in Prometheus outputs.

You can customize which attributes are exposed or hidden by using the attributes->select section in the configuration.

Common attributes table

Metrics	Attribute Name	Default
Application (all)	http.request.method	Shown
Application (all)	http.response.status_code	Shown
Application (all)	http.route	Shown if routes config exists
Application (all)	k8s.daemonset.name	Shown if Kubernetes metadata is on
Application (all)	k8s.deployment.name	Shown if Kubernetes metadata is on
Application (all)	k8s.namespace.name	Shown if Kubernetes metadata is on
Application (all)	k8s.node.name	Shown if Kubernetes metadata is on
Application (all)	k8s.owner.name	Shown if Kubernetes metadata is on
Application (all)	k8s.pod.name	Shown if Kubernetes metadata is on
Application (all)	k8s.container.name	Shown if Kubernetes metadata is on
Application (all)	k8s.pod.start_time	Shown if Kubernetes metadata is on
Application (all)	k8s.pod.uid	Shown if Kubernetes metadata is on
Application (all)	k8s.replicaset.name	Shown if Kubernetes metadata is on
Application (all)	k8s.statefulset.name	Shown if Kubernetes metadata is on
Application (all)	k8s.cluster.name	Shown if Kubernetes metadata is on
Application (all)	service.name	Resource attribute only
Application (all)	service.namespace	Resource attribute only
Application (all)	target.instance	Shown
Application (all)	url.path	Hidden
Application (client)	server.address	Hidden
Application (client)	server.port	Hidden
Application (process)	various process-specific attributes (e.g., process.command, process.executable.path)	Shown if process metrics enabled
Application RPC	various RPC attributes (rpc.method, rpc.grpc.status_code)	Shown
Application (server)	client.address	Hidden
Network	network-specific attributes (e.g., src.address, dst.address, iface)	Mostly hidden or conditional
Network (TCP failed connections)	network.tcp.handshake.role: values client or server	Shown

Note: The obi.network.inter.zone.bytes metric shares the same attribute set as obi.network.flow.bytes but all are hidden by default except k8s.cluster.name, src.zone, and dst.zone.

Note: service.name and service.namespace are no longer default metric labels. They are still reported as resource attributes (through target_info and the OTLP resource). To keep them on metric series, add them to the metric's include list under attributes->select.

Internal OBI metrics

OBI provides several internal Prometheus-format metrics to help monitor its own health and activity:

Name	Type	Description
obi_ebpf_tracer_flushes	Histogram	Size of trace groups flushed from eBPF to pipeline
obi_otel_metric_exports_total	Counter	Total metric batches sent to remote OTEL collector
obi_otel_metric_export_errors_total	CounterVec	Count of OTEL metric export errors by type
obi_otel_trace_exports_total	Counter	Total trace batches sent to remote OTEL collector
obi_otel_trace_export_errors_total	CounterVec	Count of OTEL trace export errors by type
obi_prometheus_http_requests_total	CounterVec	Prometheus scrape endpoint request count by port and path
obi_instrumented_processes	GaugeVec	Number of processes instrumented by Obi, labeled by process name
obi_instrumentation_errors_total	CounterVec	Instrumentation errors, labeled by process executable name and error type
obi_avoided_services	GaugeVec	Services OBI skipped because they already carry OpenTelemetry instrumentation, labeled by service name, namespace, and telemetry type. Emitted only when the avoided-services internal metric is enabled.
obi_internal_build_info	GaugeVec	Obi version information including build timestamp and commit ID

obi_otel_trace_exports_total and obi_otel_trace_export_errors_total reflect the real outcome reported by the exporter rather than counting attempts. Internal metrics also carry host metadata as resource attributes.

OBI can additionally report eBPF probe latency and map utilization, but that family is opt-in. It requires the ebpf metrics feature with a Prometheus endpoint, or internal_metrics.bpf_metric_scrape_interval set above zero. See the upstream BPF metrics collection reference for the metric names and labels.