Send Go runtime metrics

Open in ChatGPTOpen in Claude

This guide walks through the steps to make a Go service emit runtime metrics in the OpenTelemetry format so that they appear in the Runtime tab in the APM entity drilldown.

Use this guide if your service is Go-based but the Runtime tab shows an empty state, or if you are setting up a new Go service and want the tab populated from day one.

The go.* metric namespace

The Runtime tab reads runtime metrics in the go.* namespace emitted by the OpenTelemetry Go runtime instrumentation - for example go.goroutine.count, go.memory.used, and go.schedule.duration. A few metrics defined in the OpenTelemetry Go specification - the GC cycle count, GC pause duration, and CPU time - are not emitted by the contrib instrumentation yet, so the panels that would use them do not populate.

What you need

• A Go service running a recent Go toolchain (Go 1.22 or later recommended), already reporting to Coralogix APM.
• The OpenTelemetry go.opentelemetry.io/contrib/instrumentation/runtime package, plus the OpenTelemetry Go metric SDK and an OTLP metric exporter.
• A Coralogix Send-Your-Data API key.
• The OpenTelemetry endpoint for your Coralogix domain.
• Permission to modify the service startup code and set environment variables.

For the full base setup - Go module, traces, and the shared OpenTelemetry resource - see Golang OpenTelemetry instrumentation. This guide covers only the runtime-metrics layer on top of it.

Step 1: Install the instrumentation packages

Add the runtime instrumentation package, the OTLP metric exporter, and the metric SDK to your module.

go get \
  go.opentelemetry.io/contrib/instrumentation/runtime \
  go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc \
  go.opentelemetry.io/otel/sdk/metric \
  go.opentelemetry.io/otel/sdk/resource \
  go.opentelemetry.io/otel/semconv/v1.34.0 \
  google.golang.org/grpc/credentials

The go.opentelemetry.io/contrib/instrumentation/runtime package collects the go.* runtime metrics through the OpenTelemetry Metrics SDK, which this guide exports over OTLP to Coralogix.

Step 2: Configure the metrics exporter and meter provider

Point an OTLP metrics exporter at Coralogix. Set the connection details as environment variables so they stay out of the code.

export OTEL_EXPORTER_OTLP_ENDPOINT="<coralogix-otel-endpoint>"
export CX_PRIVATE_KEY="<CXPrivateKey>"

Replace <CXPrivateKey> with your Send-Your-Data API key and <coralogix-otel-endpoint> with the OpenTelemetry endpoint for your selected Coralogix domain. The gRPC exporter takes the endpoint as host:port without a scheme.

Then, during startup, build a MeterProvider backed by a PeriodicReader that exports through the OTLP exporter, start the runtime instrumentation, and register the scheduling-latency producer. The example below is representative - validate it against your SDK version before relying on it.

package main

import (
	"context"
	"crypto/tls"
	"os"
	"time"

	"go.opentelemetry.io/contrib/instrumentation/runtime"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/resource"
	semconv "go.opentelemetry.io/otel/semconv/v1.34.0"
	"google.golang.org/grpc/credentials"
)

// startRuntimeMetrics wires up the go.* runtime metrics and returns a shutdown
// function. Call it once during startup, before the service handles requests.
func startRuntimeMetrics(ctx context.Context) (func(context.Context) error, error) {
	// Resource correlates and groups the metrics (see Step 3).
	res, err := resource.Merge(
		resource.Default(),
		resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName("<ServiceName>"),
			semconv.K8SPodName(os.Getenv("POD_NAME")),
		),
	)
	if err != nil {
		return nil, err
	}

	// OTLP/gRPC exporter to Coralogix, reading the endpoint and key from the
	// environment variables set above.
	exp, err := otlpmetricgrpc.New(ctx,
		otlpmetricgrpc.WithEndpoint(os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")),
		otlpmetricgrpc.WithHeaders(map[string]string{
			"Authorization": "Bearer " + os.Getenv("CX_PRIVATE_KEY"),
		}),
		otlpmetricgrpc.WithTLSCredentials(credentials.NewTLS(&tls.Config{})),
	)
	if err != nil {
		return nil, err
	}

	// Register runtime.NewProducer on the reader so the scheduling-latency
	// histogram is collected (see Step 4). runtime.Start alone does not emit it.
	reader := sdkmetric.NewPeriodicReader(
		exp,
		sdkmetric.WithProducer(runtime.NewProducer()),
		sdkmetric.WithInterval(60*time.Second),
	)

	mp := sdkmetric.NewMeterProvider(
		sdkmetric.WithResource(res),
		sdkmetric.WithReader(reader),
	)
	otel.SetMeterProvider(mp)

	// Start collecting the go.* runtime metrics.
	if err := runtime.Start(runtime.WithMeterProvider(mp)); err != nil {
		return nil, err
	}

	return mp.Shutdown, nil
}

Step 3: Set the resource attributes

The Runtime tab correlates metrics to a service through the service.name resource attribute, and groups Go instances in the selector and heatmap by k8s.pod.name. When k8s.pod.name is absent, the backend falls back to other resource attributes - such as container.name, host.name, and host.id - to tell instances apart, but for pod-level granularity in the selector, set k8s.pod.name explicitly.

res, _ := resource.Merge(
	resource.Default(),
	resource.NewWithAttributes(
		semconv.SchemaURL,
		semconv.ServiceName("<ServiceName>"),
		semconv.K8SPodName(os.Getenv("POD_NAME")),
	),
)

In Kubernetes, pod-level attributes are not auto-detected by the Go SDK. To populate k8s.pod.name and k8s.namespace.name, choose one of:

• Downward API environment variables: pass pod metadata into the container as env vars and read them when building the resource. For example, set POD_NAME from the Downward API and read it with os.Getenv("POD_NAME"):

  env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name

• OpenTelemetry Operator auto-instrumentation: install the OpenTelemetry Operator and create an Instrumentation custom resource. The Operator injects the right resource attributes automatically.
• Collector-side enrichment: run the k8sattributes processor in your OpenTelemetry Collector. The processor adds Kubernetes attributes to telemetry as it passes through, so the service itself does not need to know about Kubernetes.

Without one of these, k8s.pod.name is missing and the backend distinguishes instances only by the coarser attributes it can detect, which groups multiple pods on the same node together - too coarse for per-pod debugging.

Attribute	Required	Purpose
service.name	Yes	Identifies the service in the catalog. Without it, metrics arrive but cannot be correlated to a service row.
service.namespace	Recommended	Disambiguates services with the same name in different teams or environments.
k8s.pod.name	Yes for per-pod filtering	The attribute the instances selector and heatmap group by. Set it explicitly via the Downward API, OTel Operator, or Collector enrichment for pod-level granularity.
host.name	Fallback	Used to tell instances apart when k8s.pod.name is not present (host-level, not pod-level).
deployment.environment.name	Recommended	Required for the environment filter.

Step 4: Register the scheduling-latency producer

This is the step that is easy to miss. runtime.Start() emits most of the go.* metrics - live goroutines, memory by type, allocation rate, and the GC heap goal - but it does not emit the scheduling-latency histogram on its own. That histogram is collected through a separate metric producer, which you register on the PeriodicReader with sdkmetric.WithProducer(runtime.NewProducer()).

reader := sdkmetric.NewPeriodicReader(
	exp,
	sdkmetric.WithProducer(runtime.NewProducer()), // emits the scheduling-latency histogram
)
// ...
runtime.Start(runtime.WithMeterProvider(mp)) // emits the rest of the go.* metrics

Both pieces are already present in the Step 2 snippet. If you adapt an existing meter provider instead of copying that snippet wholesale, confirm the producer is registered - otherwise every panel populates except Scheduling latency, which stays empty.

Step 5: Restart the service

Restart the service so it picks up the new dependencies and the updated startup code. On startup, confirm the service runs without build or exporter errors and that it reports no OTLP connection failures in its logs.

Step 6: Verify metrics are flowing

1. In the Coralogix toolbar, go to Explore, then Metrics.
2. Search for a go.* runtime metric - for example go.goroutine.count - filtered by your service name.
3. Confirm at least one data point has arrived in the last few minutes.

Querying by service name in Explore

The OpenTelemetry resource attribute service.name (with a dot) is transformed during ingestion to the indexed metric label serviceName (camelCase). Use serviceName when filtering metrics in Explore or PromQL queries. The Runtime tab handles this mapping automatically - you only need to know the label name when you query the metrics directly.

If go.* metrics appear but Scheduling latency stays empty, the scheduling-latency producer is not registered - return to Step 4.

If no go.* metrics arrive at all:

• Confirm startRuntimeMetrics runs on startup and that runtime.Start returns no error.
• Confirm the Authorization header contains a valid Send-Your-Data API key.
• Confirm the service has network access to the OpenTelemetry endpoint for your domain.
• Check the service logs for OTLP export errors.

Step 7: Open the Runtime tab

1. In the Coralogix toolbar, select APM.
2. Select your service.
3. Select the Runtime tab.

The widgets populate as soon as go.* metrics for the service are present in the selected time window. If you still see the empty state, return to Step 6 and confirm the metrics are reaching Coralogix.

Troubleshooting

Symptom	Likely cause	Fix
Runtime tab is hidden	The service's detected language is not Go, so the Go tab is not shown for it	Confirm the instrumentation is running and service.name matches the service in the catalog
Tab shows the empty state	No go.* metrics arriving for the service	Run through Step 6 - most often runtime.Start never ran or the exporter is misconfigured
Scheduling latency is empty but the other panels populate	runtime.NewProducer() is not registered on the reader	Add sdkmetric.WithProducer(runtime.NewProducer()) to the PeriodicReader (Step 4)
Widgets render but the per-instance filter shows only one instance	k8s.pod.name is not set, so instances cannot be told apart at pod level	Set k8s.pod.name via the Downward API, OTel Operator, or Collector enrichment
GC cycle count, GC pause duration, or CPU time panels never populate	Those metrics are not emitted by the contrib instrumentation yet	Expected - no action; the panels appear once the instrumentation adds the metrics
Metrics arrive in Explore but the catalog does not show the service	service.name is missing or differs from the spans' service name	Set service.name consistently for traces and metrics

Next steps

Read and interpret the metrics in the Go runtime metrics tab.

Related resources

• Go runtime metrics
• Runtime metrics
• Golang OpenTelemetry instrumentation
• go.opentelemetry.io/contrib/instrumentation/runtime on pkg.go.dev