Tail Sampling with OpenTelemetry using Kubernetes

This tutorial demonstrates how to configure a Kubernetes cluster and deploy OpenTelemetry to collect logs, metrics, and traces, as well as enable trace sampling.

Prerequisites

• Kubernetes cluster and app running [example found in Setup steps 1 & 2]
• Helm installed
• Coralogix account with admin access

Setup

Follow these steps if you have not yet created a Kubernetes cluster and deployed the OpenTelemetry Demo App. If you have, continue to Configuration.

STEP 1. Set up and configure a Kubernetes cluster with minikube start.

Here is an example deployment using Mac:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64
sudo install minikube-darwin-amd64 /usr/local/bin/minikube

STEP 2. Start minikube.

minikube start --memory 3000

Note: This command increases the memory to 3GB to accommodate the OpenTelemetry Demo App deployment.

STEP 2. Set up and deploy the OpenTelemetry Demo App using Docker or Kubernetes.

The example below deploys it using Kubernetes via Helm.

• Add the OpenTelemetry Helm repository:

helm repo add open-telemetry <https://open-telemetry.github.io/opentelemetry-helm-charts> 

• To install the chart with the release name my-otel-demo, run the following command:

helm install my-otel-demo open-telemetry/opentelemetry-demo 

STEP 3. Expose the front end, as well as browser spans, using port forwards:

% kubectl port-forward svc/my-otel-demo-frontendproxy 8080:8080 
% kubectl port-forward svc/my-otel-demo-otelcol 4318:4318

Validation

Ensure that the Demo App is running.

Configuration

Configure OpenTelemetry and tail sampling to send telemetry data to Coralogix.

In the following example, we configure 3 collectors and load balance them with the Coralogix-OpenTelemetry agent helm chart. It consists of a sampling policy that collects 100% of error traces and limits the remaining traces to 25%.

You will be required to input the following parameters in your configuration:

STEP 1. Update your YAML file by adding a processor and exporter with load-balancing information.

opentelemetry-collector:
  config:

   exporters:
     coralogix:
     # The Coralogix domain
     domain: "coralogix.com"
     # Your Coralogix private key is sensitive
     private_key: "MYPRIVATEKEY"

     otlp:
       # TLS configuration between the exporter LB and the exporters can be encrypted - for this environment we are instructing the exporter LB to ignore TLS errors
       endpoint: http://otel-lb:4317
       tls:
         insecure: true

   service:
     pipelines:
       traces:
          processors: [spanmetrics, batch]
          exporters: [logging, otlp]
       metrics:
         processors: [batch]
         exporters: [prometheus, logging, coralogix]
       logs:
         receivers: [otlp]
         exporters: [logging, coralogix]

Notes:

• This configuration removes the Coralogix exporter for traces, replacing it with otlp. Traces are routed only to the load balancer and apply tail sampling.
• Scale your collector to ensure that its performance isn’t degraded by a load increase as it receives traces. All available scaling policies can be found here.
• To ensure that each collector has a full view of its traces and to prevent fragmented traces, all spans with the same trace ID should be channeled to the same collector instance.
• Further details on load balancing exporter and how to use it for tracing can be found here.

STEP 2. Add the resources for trace load balancing in a new YAML file.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otelcol-tail-sampler-config
data:
  collector.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
    processors:
      attributes/shipper:
        actions:
          - key: shipper
            action: insert
            value: '${MY_POD_IP}'
      tail_sampling:
        decision_wait: 10s
        num_traces: 100
        expected_new_traces_per_sec: 10
        policies:
          [
            {
              name: errors-policy,
              type: status_code,
              status_code: {status_codes: [ERROR]}
            },
            {
              name: randomized-policy,
              type: probabilistic,
              probabilistic: {sampling_percentage: 25}
            },
          ]
    exporters:
      logging:
      coralogix:
        domain: "coralogix.com"
        private_key: "${CORALOGIX_PRIVATE_KEY}"
        application_name: "otel-coralogix"
        subsystem_name: "otel"
        application_name_attributes:
          - "applicationName"
          - "service.namespace"
          - "k8s.namespace.name"
        subsystem_name_attributes:
          - "service.name"
          - "k8s.deployment.name"
          - "k8s.statefulset.name"
          - "k8s.daemonset.name"
          - "k8s.cronjob.name"
          - "k8s.job.name"
          - "k8s.container.name"
        timeout: 30s
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [tail_sampling, attributes/shipper]
          exporters: [coralogix]
      telemetry:
        metrics:
          address: ${MY_POD_IP}:8888
        logs:
          level: "debug"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otelcol-tail-sampler
spec:
  replicas: 3
  selector:
    matchLabels:
      app: otelcol-tail-sampler
  template:
    metadata:
      labels:
        app: otelcol-tail-sampler
    spec:
      containers:
      - name: otelcol-tail-sampler
        args:
        - --config=/conf/collector.yaml
        image: otel/opentelemetry-collector-contrib:0.77.0
        env:
          - name: CORALOGIX_PRIVATE_KEY
            valueFrom:
              secretKeyRef:
                name: coralogix-keys
                key: PRIVATE_KEY
          - name: KUBE_NODE_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: spec.nodeName
          - name: MY_POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        ports:
        - containerPort: 4317
        - containerPort: 4318
        readinessProbe:
          tcpSocket:
            port: 4318
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 4318
          initialDelaySeconds: 15
          periodSeconds: 20
        resources:
          requests:
            cpu: 200m
            memory: 180Mi
          limits:
            cpu: 300m
            memory: 300M
        volumeMounts:
        - mountPath: /conf
          name: otelcol-tail-sampler-config
      volumes:
      - configMap:
          items:
          - key: collector.yaml
            path: collector.yaml
          name: otelcol-tail-sampler-config
        name: otelcol-tail-sampler-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: collector-lb-config
data:
  collector.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
    exporters:
      logging:
      loadbalancing:
        routing_key: "service"
        protocol:
          otlp:
            timeout: 4s
            tls:
              insecure: true
        resolver:
          dns:
            hostname: otelcol-tail-sampler.default.svc.cluster.local
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [loadbalancing]
      telemetry:
        metrics:
          address: ${MY_POD_IP}:8888
        logs:
          level: "debug"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otelcollector-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otelcollector-lb
  template:
    metadata:
      labels:
        app: otelcollector-lb
    spec:
      containers:
      - name: otelcollector-lb
        args:
        - --config=/conf/collector.yaml
        image: otel/opentelemetry-collector-contrib:0.77.0
        env:
          - name: KUBE_NODE_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: spec.nodeName
          - name: MY_POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        ports:
        - containerPort: 4317
        - containerPort: 4318
        readinessProbe:
          tcpSocket:
            port: 4318
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 4318
          initialDelaySeconds: 15
          periodSeconds: 20
        resources:
          requests:
            cpu: 200m
            memory: 180Mi
          limits:
            cpu: 300m
            memory: 300M
        volumeMounts:
        - mountPath: /conf
          name: collector-lb-config
      volumes:
      - configMap:
          items:
          - key: collector.yaml
            path: collector.yaml
          name: collector-lb-config
        name: collector-lb-config
---
apiVersion: v1
kind: Service
metadata:
  name: otel-lb
spec:
  selector:
    app: otelcollector-lb
  ports:
  - name: grpc
    port: 4317
    targetPort: 4317
  - name: http
    port: 4318
    targetPort: 4318
---
apiVersion: v1
kind: Service
metadata:
  name: otelcol-tail-sampler
spec:
  clusterIP: None
  selector:
    app: otelcol-tail-sampler
  ports:
    - name: grpc
      port: 4317
      targetPort: 4317
    - name: http
      port: 4318
      targetPort: 4318

Notes:

• The example file is entitled tail-sampling.yaml. Change your default namespace accordingly.
• The example creates 3 pods for tail sampling. Scale accordingly.

STEP 3. Add the resources to your Kubernetes cluster.

% kubectl apply -f tail-sampling.yaml

Validation

STEP 1. Ensure all of the pods in your cluster are running.

STEP 2. View your telemetry data in your Coralogix dashboard. Traces should arrive from the tail-sampling load balancer.

Additional Resources

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].