Kubernetes with Fluent-Bit

The following manual will help you integrate your Kubernetes logs to Coralogix using Fluent-Bit.

See other tutorials for:

• Kubernetes logs with Filebeat 
• Kubernetes logs with FluentD

Prerequisites

Before you will begin, make sure that you already have:

• Installed Kubernetes Cluster
• Enabled RBAC authorization mode support

This document includes cluster dependent URL’s. Each URL has a variable part (in Italic). Please match this part with a row entry within the following table. Copy the table row entry located under the column that matches the top level domain of your Coralogix account (.com, .in etc.). Replace the variable part of the URL with this entry.

Installation

First, you should create Kubernetes secret with Coralogix credentials:

kubectl -n kube-system create secret generic fluent-bit-coralogix-account-secrets --from-literal=PRIVATE_KEY=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

You should receive something like:

secret "fluent-bit-coralogix-account-secrets" created

If your account URL doesn’t end with .com use:

kubectl -n kube-system create secret generic fluentd-coralogix-account-secrets \ --from-literal=CORALOGIX_LOG_URL=https://api.Cluster URL/api/v1/logs \ --from-literal=CORALOGIX_TIME_DELTA_URL=https://api.Cluster URL/sdk/v1/time \ --from-literal=PRIVATE_KEY=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX \ --from-literal=APP_NAME=fluentd-coralogix-image \ --from-literal=SUB_SYSTEM=fluentd

Then you need to create fluent-bit-coralogix-logger resources on your Kubernetes cluster with these manifests:

kubectl create -f https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/fluent-bit/kubernetes/fluent-bit-coralogix-rbac.yaml
kubectl create -f https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/fluent-bit/kubernetes/fluent-bit-coralogix-svc.yaml

kubectl create -f https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/fluent-bit/kubernetes/fluent-bit-coralogix-cm.yaml
kubectl create -f https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/fluent-bit/kubernetes/fluent-bit-coralogix-ds.yaml

kubectl create -f https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/fluent-bit/kubernetes/fluent-bit-native-coralogix-cm.yaml
kubectl create -f https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/fluent-bit/kubernetes/fluent-bit-native-coralogix-ds.yaml

Output:

serviceaccount "fluent-bit-coralogix-service-account" created
clusterrole "fluent-bit-coralogix-service-account-role" created
clusterrolebinding "fluent-bit-coralogix-service-account" created
configmap "fluent-bit-coralogix-config" created
daemonset "fluent-bit-coralogix-daemonset" created
service "fluent-bit-coralogix-service" created

Now fluent-bit-coralogix-logger collects logs from your Kubernetes cluster.

Here is the example of log record:

{
    "log": "172.17.0.1 - - [05/Apr/2020:22:59:52 +0000] \"GET / HTTP/1.1\" 200 6 \"\" \"kube-probe/1.18\"\n",
    "stream": "stdout",
    "time": "2020-04-05T22:59:52.096035683Z",
    "kubernetes": {
        "pod_name": "dashboard-metrics-scraper-84bfdf55ff-l66cf",
        "namespace_name": "kubernetes-dashboard",
        "labels": {
            "k8s-app": "dashboard-metrics-scraper",
            "pod-template-hash": "84bfdf55ff"
        },
        "annotations": {
            "seccomp.security.alpha.kubernetes.io/pod": "runtime/default"
        },
        "host": "minikube",
        "container_name": "dashboard-metrics-scraper",
        "container_image": "kubernetesui/metrics-scraper:v1.0.2"
    }
}

Uninstall

If you want to remove fluent-bit-coralogix-logger from your cluster, execute this:

kubectl -n kube-system delete secret fluent-bit-coralogix-account-secrets
kubectl -n kube-system delete svc,ds,cm,clusterrolebinding,clusterrole,sa -l k8s-app=fluent-bit-coralogix-logger