This tutorial demonstrates how to configure a Kubernetes cluster and deploy OpenTelemetry to collect logs, metrics, and traces, as well as enable trace sampling.
Follow these steps if you have not yet created a Kubernetes cluster and deployed the OpenTelemetry Demo App. If you have, continue to Configuration.
STEP 1. Set up and configure a Kubernetes cluster with minikube start.
Here is an example deployment using Mac:
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 sudo install minikube-darwin-amd64 /usr/local/bin/minikube
STEP 2. Start minikube.
minikube start --memory 3000
Note: This command increases the memory to 3GB to accommodate the OpenTelemetry Demo App deployment.
STEP 2. Set up and deploy the OpenTelemetry Demo App using Docker or Kubernetes.
The example below deploys it using Kubernetes via Helm.
helm repo add open-telemetry <https://open-telemetry.github.io/opentelemetry-helm-charts>
helm install my-otel-demo open-telemetry/opentelemetry-demo
STEP 3. Expose the front end, as well as browser spans, using port forwards:
% kubectl port-forward svc/my-otel-demo-frontendproxy 8080:8080 % kubectl port-forward svc/my-otel-demo-otelcol 4318:4318
Ensure that the Demo App is running.
Configure OpenTelemetry and tail sampling to send telemetry data to Coralogix.
In the following example, we configure 3 collectors and load balance them with the Coralogix-OpenTelemetry agent helm chart. It consists of a sampling policy that collects 100% of error traces and limits the remaining traces to 25%.
You will be required to input the following parameters in your configuration:
Parameter | Description |
domain | Domain associated with your Coralogix account |
CORALOGIX_PRIVATE_KEY | Coralogix Send-Your-Data API key |
STEP 1. Update your YAML file by adding a processor and exporter with load-balancing information.
opentelemetry-collector: config: processors: tail_sampling: policies: [ { name: errors-policy, type: status_code, status_code: { status_codes: [ERROR] }, }, { name: randomized-policy, type: probabilistic, probabilistic: { sampling_percentage: 25 }, }, ] exporters: coralogix: # The Coralogix domain domain: "coralogix.com" # Your Coralogix private key is sensitive private_key: "MYPRIVATEKEY" otlp: # TLS configuration between the exporter LB and the exporters can be encrypted - for this environment we are instructing the exporter LB to ignore TLS errors endpoint: http://otel-lb:4317 tls: insecure: true service: pipelines: traces: processors: [batch] exporters: [spanmetrics, logging, otlp] metrics: processors: [batch] exporters: [prometheus, logging, coralogix] logs: receivers: [otlp] exporters: [logging, coralogix]
Notes:
STEP 2. Add the resources for trace load balancing in a new YAML file.
--- apiVersion: v1 kind: ConfigMap metadata: name: otelcol-tail-sampler-config data: collector.yaml: | receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: processors: attributes/shipper: actions: - key: shipper action: insert value: '${MY_POD_IP}' tail_sampling: decision_wait: 10s num_traces: 100 expected_new_traces_per_sec: 10 policies: [ { name: errors-policy, type: status_code, status_code: {status_codes: [ERROR]} }, { name: randomized-policy, type: probabilistic, probabilistic: {sampling_percentage: 25} }, ] exporters: logging: coralogix: domain: "coralogix.com" private_key: "${CORALOGIX_PRIVATE_KEY}" application_name: "otel-coralogix" subsystem_name: "otel" application_name_attributes: - "service.namespace" - "k8s.namespace.name" subsystem_name_attributes: - "service.name" - "k8s.deployment.name" - "k8s.statefulset.name" - "k8s.daemonset.name" - "k8s.cronjob.name" - "k8s.job.name" - "k8s.container.name" timeout: 30s service: pipelines: traces: receivers: [otlp] processors: [tail_sampling, attributes/shipper] exporters: [coralogix] telemetry: metrics: address: ${MY_POD_IP}:8888 logs: level: "debug" --- apiVersion: apps/v1 kind: Deployment metadata: name: otelcol-tail-sampler spec: replicas: 3 selector: matchLabels: app: otelcol-tail-sampler template: metadata: labels: app: otelcol-tail-sampler spec: containers: - name: otelcol-tail-sampler args: - --config=/conf/collector.yaml image: otel/opentelemetry-collector-contrib:0.77.0 env: - name: CORALOGIX_PRIVATE_KEY valueFrom: secretKeyRef: name: coralogix-keys key: PRIVATE_KEY - name: KUBE_NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: MY_POD_IP valueFrom: fieldRef: fieldPath: status.podIP ports: - containerPort: 4317 - containerPort: 4318 readinessProbe: tcpSocket: port: 4318 initialDelaySeconds: 5 periodSeconds: 10 livenessProbe: tcpSocket: port: 4318 initialDelaySeconds: 15 periodSeconds: 20 resources: requests: cpu: 200m memory: 180Mi limits: cpu: 300m memory: 300M volumeMounts: - mountPath: /conf name: otelcol-tail-sampler-config volumes: - configMap: items: - key: collector.yaml path: collector.yaml name: otelcol-tail-sampler-config name: otelcol-tail-sampler-config --- apiVersion: v1 kind: ConfigMap metadata: name: collector-lb-config data: collector.yaml: | receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: exporters: logging: loadbalancing: routing_key: "service" protocol: otlp: timeout: 4s tls: insecure: true resolver: dns: hostname: otelcol-tail-sampler.default.svc.cluster.local service: pipelines: traces: receivers: [otlp] processors: [] exporters: [loadbalancing] telemetry: metrics: address: ${MY_POD_IP}:8888 logs: level: "debug" --- apiVersion: apps/v1 kind: Deployment metadata: name: otelcollector-lb spec: replicas: 1 selector: matchLabels: app: otelcollector-lb template: metadata: labels: app: otelcollector-lb spec: containers: - name: otelcollector-lb args: - --config=/conf/collector.yaml image: otel/opentelemetry-collector-contrib:0.77.0 env: - name: KUBE_NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: MY_POD_IP valueFrom: fieldRef: fieldPath: status.podIP ports: - containerPort: 4317 - containerPort: 4318 readinessProbe: tcpSocket: port: 4318 initialDelaySeconds: 5 periodSeconds: 10 livenessProbe: tcpSocket: port: 4318 initialDelaySeconds: 15 periodSeconds: 20 resources: requests: cpu: 200m memory: 180Mi limits: cpu: 300m memory: 300M volumeMounts: - mountPath: /conf name: collector-lb-config volumes: - configMap: items: - key: collector.yaml path: collector.yaml name: collector-lb-config name: collector-lb-config --- apiVersion: v1 kind: Service metadata: name: otel-lb spec: selector: app: otelcollector-lb ports: - name: grpc port: 4317 targetPort: 4317 - name: http port: 4318 targetPort: 4318 --- apiVersion: v1 kind: Service metadata: name: otelcol-tail-sampler spec: clusterIP: None selector: app: otelcol-tail-sampler ports: - name: grpc port: 4317 targetPort: 4317 - name: http port: 4318 targetPort: 4318
Notes:
STEP 3. Add the resources to your Kubernetes cluster.
% kubectl apply -f tail-sampling.yaml
STEP 1. Ensure all of the pods in your cluster are running.
STEP 2. View your telemetry data in your Coralogix dashboard. Traces should arrive from the tail-sampling load balancer.
Need help?
Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.
Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].