Fluent Bit Tutorial: The Beginners Guide

Fluentbit-tutorial-guide

Fluent Bit is a fast and lightweight log processor, stream processor, and forwarder for Linux, OSX, Windows, and BSD family operating systems. Its focus on performance allows the collection of events from different sources and the shipping to multiple destinations without complexity. It is lightweight, allowing it to run on embedded systems as well as complex cloud-based virtual machines. In this post, we will cover the main use cases and configurations for Fluent Bit.

Fluent Bit Installation

Fluent Bit has simple installations instructions. Coralogix has a straight forward integration but if you’re not using Coralogix, then we also have instructions for Kubernetes installations.

Fluent Bit Configuration

Configuring Fluent Bit is as simple as changing a single file. You’ll find the configuration file at /fluent-bit/etc/fluent-bit.conf.

Concepts in the Fluent Bit Schema

The schema for the Fluent Bit configuration is broken down into two concepts:

  • Sections
  • Entries: Key/Value – One section may contain many Entries. An entry is a line of text that contains a Key and a Value

When writing out these concepts in your configuration file, you must be aware of the indentation requirements. Each configuration file must follow the same pattern of alignment from left to right.

The Types of Sections

The Fluent Bit configuration file supports four types of sections, each of them has a different set of available options.

  1. The Service section – defines global properties of the service
  2. The Input section – the source from where Fluent Bit can collect data
  3. The Filter section – altering the data before sending it to your destination
  4. The Output section – a destination where Fluent Bit should flush the information

Service

The Service section defines the global properties of the Fluent Bit service. This is an example of a common Service section that sets Fluent Bit to flush data to the designated output every 5 seconds with the log level set to debug. It also points Fluent Bit to the custom_parsers.conf as a Parser file.

[SERVICE]
    Flush 5
    Daemon Off
    Log_Level debug
    Parsers_File custom_parsers.conf

There are additional parameters you can set in this section. Check the documentation for more details.

Input

The INPUT section defines a source plugin. You can specify multiple inputs in a Fluent Bit configuration file. Each input is in its own INPUT section with its own configuration keys. The Name is mandatory and it lets Fluent Bit know which input plugin should be loaded. The Tag is mandatory for all plugins except for the input forward plugin (as it provides dynamic tags).

[INPUT]
    Name cpu
    Tag my_cpu

Fluent Bit supports various input plugins options. For example, if you want to tail log files you should use the Tail input plugin. There are a variety of input plugins available.

Filter

In addition to the Fluent Bit parsers, you may use filters for parsing your data. A filter plugin allows users to alter the incoming data generated by the input plugins before delivering it to the specified destination. You may use multiple filters, each one in its own FILTER section.

[FILTER]
    Name modify
    Match *
    Add user coralogix

This is a simple example for a filter that adds to each log record, from any input, the key user with the value coralogix. The Name is mandatory and it lets Fluent Bit know which filter plugin should be loaded. The Match or Match_Regex is mandatory for all plugins. If both are specified, Match_Regex takes precedence. There are lots of filter plugins to choose from.

Output

The OUTPUT section specifies a destination that certain records should follow after a Tag match. The following is a common example of flushing the logs from all the inputs to stdout.

[OUTPUT]

    Name stdout

    Match *

Similar to the INPUT and FILTER sections, the OUTPUT section requires The Name to let Fluent Bit know where to flush the logs generated by the input/s. Match or Match_Regex is mandatory as well. If both are specified, Match_Regex takes precedence. For all available output plugins.

Injecting Environment Variables into your Fluent Bit Config

Your configuration file supports reading in environment variables using the bash syntax. For example:

[OUTPUT]
    Name  ${MY_ENV_VAR}
    Match *

Breaking down your Configuration files

The @INCLUDE keyword is used for including configuration files as part of the main config, thus making large configurations more readable. You can create a single configuration file that pulls in many other files. This allows you to organize your configuration by a specific topic or action. All paths that you use will be read as relative from the root configuration file.

@INCLUDE somefile.conf
@INCLUDE someOtherFile.conf

Defining Variables

The @SET command is another way of exposing variables to Fluent Bit, used at the root level of each line in the config. This means you can not use the @SET command inside of a section. You can use this command to define variables that are not available as environment variables. They are then accessed in the exact same way.

@SET my_input=cpu

[SERVICE]
    Flush 1

[INPUT]
    Name ${my_input}

Parsing Structured and Unstructured Logs

Fluent Bit is able to capture data out of both structured and unstructured logs, by leveraging parsers. Parsers are pluggable components that allow you to specify exactly how Fluent Bit will parse your logs. For example, you can use the JSON, Regex, LTSV or Logfmt parsers. There are plenty of common parsers to choose from that come as part of the Fluent Bit installation. Parsers play a special role and must be defined inside the parsers.conf file.

[PARSER]
    Name        docker-logs
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S %z

Parsing Multi-Line Logs

If you are using tail input and your log files include multiline log lines, you should set a dedicated parser in the parsers.conf. You are then able to set the multiline configuration parameters in the main Fluent Bit configuration file. Multiline logs are a common problem with Fluent Bit and we have written some documentation to support our users.

Fluent Bit Configuration Examples

We have included some examples of useful Fluent Bit configuration files that showcase a specific use case. Each file will use the components that have been listed in this article and should serve as concrete examples of how to use these features.

Example 1 – Simple Tail Input to Coralogix

The Chosen application name is “prod” and the subsystem is “app”, you may later filter logs based on these metadata fields.

[SERVICE]
    # setting application logs level to debug
    Log_Level debug
    # setting location for the parsers file
    Parsers_File parsers.conf
    # setting location for Coralogix plugin
    Plugins_File  plugins.conf
[INPUT]
    # using tail input to tail log files
    Name        tail
    # setting the path for the file to tail
    Path        /var/log/fluent-bit/app.log
    # enabling multiline options
    Multiline On
    # using the parser called multiline_pattern for determine first line
    Parser_Firstline multiline_pattern
    # Adding a field named filename to be assigned with the path of the tailed file
    Path_Key filename
[OUTPUT]
    # setting up Coralogix's output plugin to forward logs to Coralogix
    Name coralogix
    # here you add your Coralogix account private key
    Private_Key xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    # here you add your Coralogix account company ID
    company_id xxxx
    # setting application name as prod
    App_Name prod
    # setting subsystem name as app
    Sub_Name app
    # forward all logs from all inputs
    Match *

The parsers file includes only one parser, which is used to tell Fluent Bit where the beginning of a line is. It was built to match a beginning of a line as written in our tailed file, e.g.

[2020-03-12 14:14:55] .....
[PARSER]
    Name	multiline_pattern
    Format	regex
    Regex  	^\[(?<timestamp>[0-9]{2,4}\-[0-9]{1,2}\-[0-9]{1,2} [0-9]{1,2}\:[0-9]{1,2}\:[0-9]{1,2})\] (?<message>.*)

Any other line which does not start similar to the above will be appended to the former line. This parser also divides the text into 2 fields, timestamp and message, to form a JSON entry where the timestamp field will possess the actual log timestamp, e.g. 2020-03-12 14:14:55, and Fluent Bit places the rest of the text into the message field.

Example 2 – Kubernetes Integration

Here we can see a Kubernetes Integration. This time, rather than editing a file directly, we need to define a ConfigMap to contain our configuration:

kind: ConfigMap
apiVersion: v1
metadata:
  name: fluent-bit-coralogix-config
  namespace: kube-system
  labels:
    k8s-app: fluent-bit-coralogix-logger
data:
  fluent-bit.conf: |-
    [SERVICE]
        # flush timeout to define when is required to flush the records ingested by input plugins through the defined output plugins
        Flush         1
        # boolean value to set if Fluent Bit should run as a Daemon (background) or not
        Daemon        Off
        # setting application logs level to warning
        Log_Level     warning
        # enable built-in HTTP Server
        HTTP_Server   On
        # setting location for the parsers file
        Parsers_File  parsers.conf
        # setting location for Coralogix plugin
        Plugins_File  plugins.conf
    [INPUT]
        # using tail input to tail log files
        Name              tail
        # tagging the input
        Tag               kube.*
 
        # setting the paths for the kubernetes cluster files to tail
        Path              /var/log/containers/*.log
        # specify the name of a parser to interpret the entry as a structured message
        Parser            docker
        # specify the database file to keep track of monitored files and offsets
        DB                /var/log/flb_kube.db
        # set a limit of memory that Tail plugin can use when appending data to the Engine.
        # if the limit is reach, it will be paused; when the data is flushed it resumes
        Mem_Buf_Limit     5MB
        # when a monitored file reach it buffer capacity due to a very long line (Buffer_Max_Size), the default behavior is to stop monitoring that file.
        # skip_Long_Lines alter that behavior and instruct Fluent Bit to skip long lines and continue processing other lines that fits into the buffer size
        Skip_Long_Lines   On
        # the interval of refreshing the list of watched files in seconds
        Refresh_Interval  10
    [FILTER]
        # name of the filter plugin
        Name                kubernetes
        # a pattern to match against the tags of incoming records
        Match               kube.*
        # allow Kubernetes Pods to exclude their logs from the log processor
        K8S-Logging.Exclude On
    [OUTPUT]
        # setting up Coralogix's output plugin to forward logs to Coralogix
        Name          coralogix
        # a pattern to match against the tags of incoming records
        Match         kube.*
        # coralogix account private key, which already exists after creating Kubernetes secret with Coralogix credentials
        Private_Key   ${PRIVATE_KEY}
        # setting kubernetes.namespace_name as the application name in Coralogix, dynamically 
        App_Name_Key  kubernetes.namespace_name
        # setting kubernetes.container_name as the subsystem name in Coralogix, dynamically 
        Sub_Name_Key  kubernetes.container_name
        # using the log time field as timestamp in Coralogix
        Time_Key      time
        # this option is getting the hostname from field kubernetes.host. If its not present, then hostname will be detected automatically
        Host_Key      kubernetes.host
        # integer value to set the maximum number of retries allowed
        Retry_Limit   5

That’s a wrap

We’ve gone through the basic concepts involved in Fluent Bit. It is a very powerful and flexible tool, and when combined with Coralogix, you can easily pull your logs from your infrastructure and develop new, actionable insights that will improve your observability and speed up your troubleshooting.

Start solving your production issues faster

Let's talk about how Coralogix can help you better understand your logs

Managed, Scaled and Compliant ELK Stack

No credit card required

Get a personalized demo

Jump on a call with one of our experts and get a live personalized demonstration