# Send Go runtime metrics Copy as Markdown[Open in ChatGPT](https://chatgpt.com/?q=Read%20https%3A%2F%2Fcoralogix.com%2Fdocs%2Fuser-guides%2Fapm-v2%2Ffeatures%2Fruntime-metrics%2Fsend-go-metrics.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.)[Open in Claude](https://claude.ai/new?q=Read%20https%3A%2F%2Fcoralogix.com%2Fdocs%2Fuser-guides%2Fapm-v2%2Ffeatures%2Fruntime-metrics%2Fsend-go-metrics.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.) This guide walks through the steps to make a Go service emit runtime metrics in the OpenTelemetry format so that they appear in the [Runtime tab](https://coralogix.com/docs/user-guides/apm-v2/features/runtime-metrics/go.md) in the APM entity drilldown. Use this guide if your service is Go-based but the Runtime tab shows an empty state, or if you are setting up a new Go service and want the tab populated from day one. The go.\* metric namespace The Runtime tab reads runtime metrics in the `go.*` namespace emitted by the OpenTelemetry Go runtime instrumentation - for example `go.goroutine.count`, `go.memory.used`, and `go.schedule.duration`. A few metrics defined in the OpenTelemetry Go specification - the GC cycle count, GC pause duration, and CPU time - are not emitted by the contrib instrumentation yet, so the panels that would use them do not populate. ## What you need[​](#what-you-need "Direct link to What you need") * A Go service running a recent Go toolchain (Go 1.22 or later recommended), already reporting to Coralogix [APM](https://coralogix.com/docs/user-guides/apm-v2/getting-started/apm-onboarding-tutorial.md). * The OpenTelemetry [`go.opentelemetry.io/contrib/instrumentation/runtime`](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/runtime) package, plus the OpenTelemetry Go metric SDK and an OTLP metric exporter. * A Coralogix Send-Your-Data API key. * The OpenTelemetry endpoint for your Coralogix domain. * Permission to modify the service startup code and set environment variables. For the full base setup - Go module, traces, and the shared OpenTelemetry resource - see [Golang OpenTelemetry instrumentation](https://coralogix.com/docs/opentelemetry/instrumentation-options/golang-opentelemetry-instrumentation.md). This guide covers only the runtime-metrics layer on top of it. ## Step 1: Install the instrumentation packages[​](#step-1-install-the-instrumentation-packages "Direct link to Step 1: Install the instrumentation packages") Add the runtime instrumentation package, the OTLP metric exporter, and the metric SDK to your module. ``` go get \ go.opentelemetry.io/contrib/instrumentation/runtime \ go.opentelemetry.io/otel \ go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc \ go.opentelemetry.io/otel/sdk/metric \ go.opentelemetry.io/otel/sdk/resource \ go.opentelemetry.io/otel/semconv/v1.34.0 \ google.golang.org/grpc/credentials ``` The `go.opentelemetry.io/contrib/instrumentation/runtime` package collects the `go.*` runtime metrics through the OpenTelemetry Metrics SDK, which this guide exports over OTLP to Coralogix. ## Step 2: Configure the metrics exporter and meter provider[​](#step-2-configure-the-metrics-exporter-and-meter-provider "Direct link to Step 2: Configure the metrics exporter and meter provider") Point an OTLP metrics exporter at Coralogix. Set the connection details as environment variables so they stay out of the code. ``` export OTEL_EXPORTER_OTLP_ENDPOINT="" export CX_PRIVATE_KEY="" ``` Replace `` with your Send-Your-Data API key and `` with the OpenTelemetry endpoint for your selected Coralogix domain. The gRPC exporter takes the endpoint as `host:port` without a scheme. Then, during startup, build a `MeterProvider` backed by a `PeriodicReader` that exports through the OTLP exporter, start the runtime instrumentation, and register the scheduling-latency producer. The example below is representative - validate it against your SDK version before relying on it. ``` package main import ( "context" "crypto/tls" "os" "time" "go.opentelemetry.io/contrib/instrumentation/runtime" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc" sdkmetric "go.opentelemetry.io/otel/sdk/metric" "go.opentelemetry.io/otel/sdk/resource" semconv "go.opentelemetry.io/otel/semconv/v1.34.0" "google.golang.org/grpc/credentials" ) // startRuntimeMetrics wires up the go.* runtime metrics and returns a shutdown // function. Call it once during startup, before the service handles requests. func startRuntimeMetrics(ctx context.Context) (func(context.Context) error, error) { // Resource correlates and groups the metrics (see Step 3). res, err := resource.Merge( resource.Default(), resource.NewWithAttributes( semconv.SchemaURL, semconv.ServiceName(""), semconv.K8SPodName(os.Getenv("POD_NAME")), ), ) if err != nil { return nil, err } // OTLP/gRPC exporter to Coralogix, reading the endpoint and key from the // environment variables set above. exp, err := otlpmetricgrpc.New(ctx, otlpmetricgrpc.WithEndpoint(os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")), otlpmetricgrpc.WithHeaders(map[string]string{ "Authorization": "Bearer " + os.Getenv("CX_PRIVATE_KEY"), }), otlpmetricgrpc.WithTLSCredentials(credentials.NewTLS(&tls.Config{})), ) if err != nil { return nil, err } // Register runtime.NewProducer on the reader so the scheduling-latency // histogram is collected (see Step 4). runtime.Start alone does not emit it. reader := sdkmetric.NewPeriodicReader( exp, sdkmetric.WithProducer(runtime.NewProducer()), sdkmetric.WithInterval(60*time.Second), ) mp := sdkmetric.NewMeterProvider( sdkmetric.WithResource(res), sdkmetric.WithReader(reader), ) otel.SetMeterProvider(mp) // Start collecting the go.* runtime metrics. if err := runtime.Start(runtime.WithMeterProvider(mp)); err != nil { return nil, err } return mp.Shutdown, nil } ``` ## Step 3: Set the resource attributes[​](#step-3-set-the-resource-attributes "Direct link to Step 3: Set the resource attributes") The Runtime tab correlates metrics to a service through the `service.name` resource attribute, and groups Go instances in the selector and heatmap by `k8s.pod.name`. When `k8s.pod.name` is absent, the backend falls back to other resource attributes - such as `container.name`, `host.name`, and `host.id` - to tell instances apart, but for pod-level granularity in the selector, set `k8s.pod.name` explicitly. ``` res, _ := resource.Merge( resource.Default(), resource.NewWithAttributes( semconv.SchemaURL, semconv.ServiceName(""), semconv.K8SPodName(os.Getenv("POD_NAME")), ), ) ``` In Kubernetes, pod-level attributes are not auto-detected by the Go SDK. To populate `k8s.pod.name` and `k8s.namespace.name`, choose one of: * **Downward API environment variables**: pass pod metadata into the container as env vars and read them when building the resource. For example, set `POD_NAME` from the Downward API and read it with `os.Getenv("POD_NAME")`: ``` env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name ``` * **OpenTelemetry Operator auto-instrumentation**: install the OpenTelemetry Operator and create an `Instrumentation` custom resource. The Operator injects the right resource attributes automatically. * **Collector-side enrichment**: run the [`k8sattributes`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor) processor in your OpenTelemetry Collector. The processor adds Kubernetes attributes to telemetry as it passes through, so the service itself does not need to know about Kubernetes. Without one of these, `k8s.pod.name` is missing and the backend distinguishes instances only by the coarser attributes it can detect, which groups multiple pods on the same node together - too coarse for per-pod debugging. | Attribute | Required | Purpose | | ----------------------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `service.name` | Yes | Identifies the service in the catalog. Without it, metrics arrive but cannot be correlated to a service row. | | `service.namespace` | Recommended | Disambiguates services with the same name in different teams or environments. | | `k8s.pod.name` | Yes for per-pod filtering | The attribute the instances selector and heatmap group by. Set it explicitly via the Downward API, OTel Operator, or Collector enrichment for pod-level granularity. | | `host.name` | Fallback | Used to tell instances apart when `k8s.pod.name` is not present (host-level, not pod-level). | | `deployment.environment.name` | Recommended | Required for the environment filter. | ## Step 4: Register the scheduling-latency producer[​](#step-4-register-the-scheduling-latency-producer "Direct link to Step 4: Register the scheduling-latency producer") This is the step that is easy to miss. `runtime.Start()` emits most of the `go.*` metrics - live goroutines, memory by type, allocation rate, and the GC heap goal - but it does **not** emit the scheduling-latency histogram on its own. That histogram is collected through a separate metric producer, which you register on the `PeriodicReader` with `sdkmetric.WithProducer(runtime.NewProducer())`. ``` reader := sdkmetric.NewPeriodicReader( exp, sdkmetric.WithProducer(runtime.NewProducer()), // emits the scheduling-latency histogram ) // ... runtime.Start(runtime.WithMeterProvider(mp)) // emits the rest of the go.* metrics ``` Both pieces are already present in the Step 2 snippet. If you adapt an existing meter provider instead of copying that snippet wholesale, confirm the producer is registered - otherwise every panel populates except **Scheduling latency**, which stays empty. ## Step 5: Restart the service[​](#step-5-restart-the-service "Direct link to Step 5: Restart the service") Restart the service so it picks up the new dependencies and the updated startup code. On startup, confirm the service runs without build or exporter errors and that it reports no OTLP connection failures in its logs. ## Step 6: Verify metrics are flowing[​](#step-6-verify-metrics-are-flowing "Direct link to Step 6: Verify metrics are flowing") 1. In the Coralogix toolbar, go to **Explore**, then **Metrics**. 2. Search for a `go.*` runtime metric - for example `go.goroutine.count` - filtered by your service name. 3. Confirm at least one data point has arrived in the last few minutes. Querying by service name in Explore The OpenTelemetry resource attribute `service.name` (with a dot) is transformed during ingestion to the indexed metric label `serviceName` (camelCase). Use `serviceName` when filtering metrics in Explore or PromQL queries. The Runtime tab handles this mapping automatically - you only need to know the label name when you query the metrics directly. If `go.*` metrics appear but **Scheduling latency** stays empty, the scheduling-latency producer is not registered - return to Step 4. If no `go.*` metrics arrive at all: * Confirm `startRuntimeMetrics` runs on startup and that `runtime.Start` returns no error. * Confirm the `Authorization` header contains a valid Send-Your-Data API key. * Confirm the service has network access to the OpenTelemetry endpoint for your domain. * Check the service logs for OTLP export errors. ## Step 7: Open the Runtime tab[​](#step-7-open-the-runtime-tab "Direct link to Step 7: Open the Runtime tab") 1. In the Coralogix toolbar, select **APM**. 2. Select your service. 3. Select the **Runtime** tab. The widgets populate as soon as `go.*` metrics for the service are present in the selected time window. If you still see the empty state, return to Step 6 and confirm the metrics are reaching Coralogix. ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") | Symptom | Likely cause | Fix | | -------------------------------------------------------------------- | ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | Runtime tab is hidden | The service's detected language is not Go, so the Go tab is not shown for it | Confirm the instrumentation is running and `service.name` matches the service in the catalog | | Tab shows the empty state | No `go.*` metrics arriving for the service | Run through Step 6 - most often `runtime.Start` never ran or the exporter is misconfigured | | **Scheduling latency** is empty but the other panels populate | `runtime.NewProducer()` is not registered on the reader | Add `sdkmetric.WithProducer(runtime.NewProducer())` to the `PeriodicReader` (Step 4) | | Widgets render but the per-instance filter shows only one instance | `k8s.pod.name` is not set, so instances cannot be told apart at pod level | Set `k8s.pod.name` via the Downward API, OTel Operator, or Collector enrichment | | GC cycle count, GC pause duration, or CPU time panels never populate | Those metrics are not emitted by the contrib instrumentation yet | Expected - no action; the panels appear once the instrumentation adds the metrics | | Metrics arrive in Explore but the catalog does not show the service | `service.name` is missing or differs from the spans' service name | Set `service.name` consistently for traces and metrics | ## Next steps[​](#next-steps "Direct link to Next steps") Read and interpret the metrics in the [Go runtime metrics](https://coralogix.com/docs/user-guides/apm-v2/features/runtime-metrics/go.md) tab. ## Related resources[​](#related-resources "Direct link to Related resources") * [Go runtime metrics](https://coralogix.com/docs/user-guides/apm-v2/features/runtime-metrics/go.md) * [Runtime metrics](https://coralogix.com/docs/user-guides/apm-v2/features/runtime-metrics.md) * [Golang OpenTelemetry instrumentation](https://coralogix.com/docs/opentelemetry/instrumentation-options/golang-opentelemetry-instrumentation.md) * [`go.opentelemetry.io/contrib/instrumentation/runtime` on pkg.go.dev](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/runtime)