Our next-gen architecture is built to help you make sense of your ever-growing data Watch a 4-min demo video!

Back to All Integrations

Fluent Bit Fluent Bit

Last Updated: Sep. 12, 2022

Coralogix provides seamless integration with Fluent-Bit so you can send your logs from anywhere and parse them according to your needs.

Prerequisites

Usage

You must provide the following four variables when creating a Coralogix logger instance.

Private Key –  A unique ID that represents your Coralogix team. The private key can be found under ‘Data Flow’->’API Keys’. Use the ‘Send Your Logs’ key at the bottom. Please note that the key should be inserted without quotation marks or apostrophes.

Application Name – The name of your main application, for example, a company named “SuperData” would probably insert the “SuperData” string parameter or if they want to debug their test environment they might insert the “SuperData– Test”.

SubSystem Name – Your application probably has multiple subsystems, for example, Backend servers, Middleware, Frontend servers, etc. in order to help you examine the data you need, inserting the subsystem parameter is vital.

HOST – Depending on your Coralogix account geolocation, you will need to choose the correct URL (table with all the clusters can be found below).

Configuration

Open your Fluent-Bit configuration file and add the following:

[FILTER]
        Name        nest
        Match       *
        Operation   nest
        Wildcard    *
        Nest_under  text
[FILTER]
        Name    modify
        Match   *
        Add    applicationName APP
        Add    subsystemName SUB_SYS
        Add    computerName ${HOSTNAME}
[OUTPUT]
        Name                  http
        Match                 *
        Host                  <CLUSTER_URL>
        Port                  443
        URI                   /logs/rest/singles
        Format                json_lines
        TLS                   On
        Header                private_key XXX
        compress              gzip
        Retry_Limit           10
[OUTPUT]
        name      stdout
        match     *
        format    json_lines

The first three keys (Private_key,App_nameSub_name) are mandatory.

This document includes cluster dependent URL’s. Please match the last few characters of your Coralogix Team’s name URL (.com, .eu2.coralogix.com, .us, .in, etc.) to the table below, to determine the correct URL to use for the “Host” field in the [output] section of the Fluent Bit configuration.

ClusterAPI Endpoint
.comhttps://api.coralogix.com
.ushttps://api.coralogix.us
.inhttps://api.app.coralogix.in
.app.eu2.coralogix.comhttps://api.eu2.coralogix.com
.app.coralogixsg.comhttps://api.coralogixsg.com

Application and subsystem name

If you wish to set your application and subsystem names to a fixed value, in your config file use App_Name and Sub_Name as described above. In case your input stream is a JSON object, you can extract APP_NAME and/or SUB_NAME from the JSON using the App_Name_Key and Sub_Name_Key options in your config file instead of App_Name and Sub_Name:

App_Name_Key APP_NAME_KEY
Sub_Name_Key SUB_NAME_KEY

For instance, with the bellow JSON App_Name_Key application will extract “testApp” into Coralogix applicationName.

{
    "application": "testApp",
    "subsystem": "testSub",
    "code": "200",
    "stream": "stdout",
    "timestamp": "2016-07-20T17:05:17.743Z",
    "message": "hello_world",
}

*Note – nested JSONs are also supported so you can extract into App_Name_Key and/or Sub_Name_Key nested values, e.g. App_Name_Key log.application.

Record content

In case your input stream is a JSON object and you don’t want to send the entire JSON, rather just a portion of it, you can add the Log_Key parameter, in your Fluent-Bit configuration file–>output section, with the name of the key you want to send. For instance, with the above example, if you write:

Log_Key message

then only the key message will be sent. If you do want to send the entire JSON then you can just delete this parameter from your configuration file.

Timestamp

If you want to use some field as timestamp in Coralogix, you can use Time_Key option:

Time_Key timestamp

then you will see that logs records have timestamp from this field.

Note: We accept only logs that are not older than 24 hours.

Run

Docker

Build a Docker image with your fluent-bit.conf:

If your Coralogix account top-level domain is different than ‘.com’ set the following environment variable

CORALOGIX_LOG_URL=https://api.Cluster URL/logs/rest/singles
FROM golang:alpine AS builder
RUN apk add --no-cache gcc libc-dev git
WORKDIR /go/src/app
RUN wget https://raw.githubusercontent.com/fluent/fluent-bit/master/conf/plugins.conf && \
    echo "    Path /fluent-bit/plugins/out_coralogix.so" | tee -a plugins.conf
RUN go mod init
RUN wget https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/fluent-bit/plugin/out_coralogix.go && \
    go get . && \
    go build -buildmode=c-shared -o out_coralogix.so .


FROM fluent/fluent-bit:1.4
MAINTAINER Coralogix Inc. <[email protected]>
LABEL Description="Special Fluent-Bit image for Coralogix integration" Vendor="Coralogix Inc." Version="1.0.0"
COPY --from=builder /lib/libc.musl-x86_64.so* /lib/x86_64-linux-gnu/
COPY --from=builder /go/src/app/out_coralogix.so /fluent-bit/plugins/
COPY --from=builder /go/src/app/plugins.conf /fluent-bit/etc/
COPY fluent-bit.conf /fluent-bit/etc/

Before deploying your container don’t forget to mount volume with your logs.

Development

Requirements

  • Linux x64
  • Go version >= 1.11.x

Sources

You can download sources here.

Build

$ cd plugin/
$ make

Auto-mapping support

In case your raw log message is a JSON object containing fields with information such as geographic location (lat, lon), DateTime, or Ip address, you may change and add a specific suffix (see followed examples) to the key name using a filter in your configuration (or by using Coralogix parsing rules) so the same field will be automatically mapped as geo-pointdateIP respectively. As a result, you will be able to create a geo-location map visualization, use your log timestamp as the timestamp in range queries, and with Kibana visualization and query IP addresses using the CIDR notation.

E.g. Geographic location

Original log

{
  ...
  "text": "Geo-point data",
  "location": { 
    "lat": 41.12,
    "lon": -71.34
  }
  ...
}

Adding _geopoint suffix to the location object name

{
  ...
  "text": "Geo-point data",
  "location_geopoint": { 
    "lat": 41.12,
    "lon": -71.34
  },
  ...
}

E.g. DateTime

Original log

{
  ...
  "time": "2020-10-13T09:45:33.783441Z",
  ...
}

Adding _custom_timestamp suffix to the time key name

{
 ... 
 "time_custom_timestamp": "2020-10-13T09:45:33.783441Z",
 ... 
}

Note that the time format must be date_optional_time or strict_date_optional_time.

E.g. Ip

Original log

{
  ...
  "ip_addr": "192.168.1.1",
  ...
}

Adding _ipaddr suffix to the location object name

{
  ...
  "ip_addr_ipaddr": "192.168.1.1",
  ...
}

If you need additional support or have any questions, our support team is available to you 24/7 via our in-app chat.

On this page