Learn more about Streama© – the foundational technology behind our stateful streaming data platform. Learn More

Kubernetes with Fluentd

The following manual will help you integrate Coralogix logging into your Kubernetes cluster using Fluentd.

See other tutorials for:

Prerequisites

This document includes cluster dependent URL’s. Each URL has a variable part (in Italic). Please match this part with a row entry within the following table. Copy the table row entry located under the column that matches the top level domain of your Coralogix account (.com, .in etc.). Replace the variable part of the URL with this entry.

 .com.us.in
Elasticsearch-APIhttps://coralogix-esapi.coralogix.com:9443https://esapi.coralogix.us:9443https://es-api.app.coralogix.in:9443
SSL Certificateshttps://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/Coralogix-EU.crthttps://www.amazontrust.com/repository/AmazonRootCA1.pemhttps://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/Coralogix-IN
.pem
coralogix.comcoralogix.usapp.coralogix.in

Before you begin, make sure you have:

  • An installed Kubernetes Cluster
  • Enabled RBAC authorization mode
  • Pull the image coralogixrepo/fluentd-coralogix-image:latest
  • Private key – you can find it in your Coralogix account at Settings –> Send your logs –> upper left corner

Installation

First, you should create a Kubernetes secret:

If your Coralogix account name ends with .com use:

kubectl -n kube-system create secret generic fluentd-coralogix-account-secrets --from-literal=PRIVATE_KEY=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX --from-literal=APP_NAME=fluentd-coralogix-image --from-literal=SUB_SYSTEM=fluentd

If your Coralogix account top level domain is not ‘.com’ use:

kubectl -n kube-system create secret generic fluentd-coralogix-account-secrets \ –from-literal=CORALOGIX_LOG_URL=https://api.Cluster URL/api/v1/logs \ –from-literal=CORALOGIX_TIME_DELTA_URL=https://api.Cluster URL/sdk/v1/time \ –from-literal=PRIVATE_KEY=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX \ –from-literal=APP_NAME=fluentd-coralogix-image \ –from-literal=SUB_SYSTEM=fluentd

You should have something like:

secret "fluentd-coralogix-account-secrets" created

Then you will need to apply our fluentd-coralogix-logger to your Kubernetes cluster:

kubectl create -f https://raw.githubusercontent.com/coralogix/fluentd-coralogix-image/master/examples/kubernetes/fluentd-coralogix-rbac.yaml
kubectl create -f https://raw.githubusercontent.com/coralogix/fluentd-coralogix-image/master/examples/kubernetes/fluentd-coralogix-cm.yaml
kubectl create -f https://raw.githubusercontent.com/coralogix/fluentd-coralogix-image/master/examples/kubernetes/fluentd-coralogix-ds.yaml
kubectl create -f https://raw.githubusercontent.com/coralogix/fluentd-coralogix-image/master/examples/kubernetes/fluentd-coralogix-svc.yaml

Output:

serviceaccount "fluentd-coralogix-service-account" created
clusterrole "fluentd-coralogix-service-account-role" created
clusterrolebinding "fluentd-coralogix-service-account" created
configmap "fluentd-coralogix-configs" created
daemonset "fluentd-coralogix-daemonset" created
service "fluentd-coralogix-service" created

Now, fluentd-coralogix-logger is collecting logs from your Kubernetes cluster.

Usage

You can view some usage examples here.

Update

Once installed, the deployment will not automatically update. In order to update it, you need to delete the deployment’s pods and wait for it to be recreated. After recreation, it should use the latest image.

Delete all fluentd-coralogix-logger pods:

kubectl -n kube-system delete $(kubectl -n kube-system get pod -o name | grep "fluentd-coralogix-daemonset")

Note that if you would like to send all of the log content with Kubernetes metadata like labels, tags, pod name, etc. you should remove log_key_name log from Fluentd.conf.

Uninstall

If you want to remove fluentd-coralogix-logger from your cluster, execute this:

kubectl -n kube-system delete secret fluentd-coralogix-account-secrets
kubectl -n kube-system delete svc,ds,cm,clusterrolebinding,clusterrole,sa -l k8s-app=fluentd-coralogix-logger