Coralogix provides seamless integration with Fluent Bit so you can send your logs from anywhere and parse them according to your needs.
Note! Coralogix supports Fluent Bit v2.0.7 and onwards.
You must provide the following four variables when creating a Coralogix logger instance.
Private Key – Your private key is a unique ID that represents your Coralogix team. Input the key without quotation marks or apostrophes.
Application Name – The name of your application, as it will appear in your Coralogix dashboard. For example, a company named SuperData might insert the SuperData
string parameter. If SuperData wants to debug its test environment, it might use SuperData–Test
.
SubSystem Name – The name of your subsystem, as it will appear in your Coralogix dashboard. Applications often have multiple subsystems (ie. Backend Servers, Middleware, Frontend Servers, etc.). In order to help you examine the data you need, inserting the subsystem parameter is vital.
HOST – Find the endpoint here matching your Coralogix domain.
Open your Fluent-Bit
configuration file and add the following:
[FILTER]
Name nest
Match *
Operation nest
Wildcard *
Nest_under text
[FILTER]
Name modify
Match *
Add applicationName APP
Add subsystemName SUB_SYS
Add computerName ${HOSTNAME}
[OUTPUT]
Name http
Match *
Host <CLUSTER_HOST_NAME>
Port 443
URI /logs/rest/singles
Format json_lines
TLS On
Header private_key XXX
compress gzip
Retry_Limit 10
[OUTPUT]
name stdout
match *
format json_lines
The first three keys (Private_key
,App_name
, Sub_name
) are mandatory.
If you wish to set your application and subsystem names as a fixed value, use App_Name and Sub_Name as described above in your configuration file. In case your input stream is a JSON
object, you can extract APP_NAME and/or SUB_NAME from the JSON
using the App_Name_Key
and Sub_Name_Key
options in your config file instead of App_Name
and Sub_Name
:
App_Name_Key APP_NAME_KEY
Sub_Name_Key SUB_NAME_KEY
For instance, with the bellow JSON
App_Name_Key application
will extract “testApp” into Coralogix applicationName.
{
"application": "testApp",
"subsystem": "testSub",
"code": "200",
"stream": "stdout",
"timestamp": "2016-07-20T17:05:17.743Z",
"message": "hello_world",
}
*Note – nested JSONs are also supported so you can extract into App_Name_Key
and/or Sub_Name_Key
nested values, e.g. App_Name_Key log.application
.
In case your input stream is a JSON
object and you don’t want to send the entire JSON
, rather just a portion of it, you can add the Log_Key parameter, in your Fluent-Bit
configuration file–>output section, with the name of the key you want to send. For instance, with the above example, if you write:
Log_Key message
then only the key message
will be sent. If you do want to send the entire JSON
then you can just delete this parameter from your configuration file.
If you want to use some field as timestamp
in Coralogix, you can use Time_Key option:
Time_Key timestamp
then you will see that logs records have timestamp from this field.
Note: We accept only logs that are not older than 24 hours.
Build a Docker image with your fluent-bit.conf:
Insert your Coralogix domain in the endpoint as follows:
CORALOGIX_LOG_URL=https://ingress.<domain>/logs/rest/singles
FROM golang:alpine AS builder
RUN apk add --no-cache gcc libc-dev git
WORKDIR /go/src/app
RUN wget https://raw.githubusercontent.com/fluent/fluent-bit/master/conf/plugins.conf && \
echo " Path /fluent-bit/plugins/out_coralogix.so" | tee -a plugins.conf
RUN go mod init
RUN wget https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/fluent-bit/plugin/out_coralogix.go && \
go get . && \
go build -buildmode=c-shared -o out_coralogix.so .
FROM fluent/fluent-bit:1.4
MAINTAINER Coralogix Inc. <[email protected]>
LABEL Description="Special Fluent-Bit image for Coralogix integration" Vendor="Coralogix Inc." Version="1.0.0"
COPY --from=builder /lib/libc.musl-x86_64.so* /lib/x86_64-linux-gnu/
COPY --from=builder /go/src/app/out_coralogix.so /fluent-bit/plugins/
COPY --from=builder /go/src/app/plugins.conf /fluent-bit/etc/
COPY fluent-bit.conf /fluent-bit/etc/
Before deploying your container don’t forget to mount volume with your logs.
Linux
x64Go
version >= 1.11.xYou can download sources here.
$ cd plugin/ $ make
In case your raw log message is a JSON object containing fields with information such as geographic location (lat, lon), DateTime, or Ip address, you may change and add a specific suffix (see followed examples) to the key name using a filter in your configuration (or by using Coralogix parsing rules) so the same field will be automatically mapped as geo-point, date, IP respectively. As a result, you will be able to create a geo-location map visualization, use your log timestamp as the timestamp in range queries, and with Kibana visualization and query IP addresses using the CIDR notation.
E.g. Geographic location
Original log
{
...
"text": "Geo-point data",
"location": {
"lat": 41.12,
"lon": -71.34
}
...
}
Adding _geopoint suffix to the location object name
{
...
"text": "Geo-point data",
"location_geopoint": {
"lat": 41.12,
"lon": -71.34
},
...
}
E.g. DateTime
Original log
{
...
"time": "2020-10-13T09:45:33.783441Z",
...
}
Adding _custom_timestamp suffix to the time key name
{
...
"time_custom_timestamp": "2020-10-13T09:45:33.783441Z",
...
}
Note that the time format must be date_optional_time
or strict_date_optional_time
.
E.g. Ip
Original log
{
...
"ip_addr": "192.168.1.1",
...
}
Adding _ipaddr suffix to the location object name
{
...
"ip_addr_ipaddr": "192.168.1.1",
...
}
If you need additional support or have any questions, our support team is available to you 24/7 via our in-app chat.