A practical guide to FluentD

coralogix fluentd guide

In this post we will cover some of the main use cases FluentD supports and provides example FluentD configurations for the different cases.

What is Fluentd

Fluentd is an open source data collector, which allows you to unify your data collection and consumption. Fluentd was conceived by Sadayuki “Sada” Furuhashi in 2011. Sada is a co-founder of Treasure Data, Inc., the primary sponsor of the Fluentd and the source of stable Fluentd releases.

Installation

Fluentd installation instructions can be found on the fluentd website.

Here are Coralogix’s Fluentd plugin installation instructions

Coralogix also has a Fluentd plug-in image for k8s.  This doc describes Coralogix integration with Kubernetes. 

Configuration

Flunetd configuration file consists of the following directives (only source and match are mandatory ones):

  1. source directives determine the input sources.
  2. match directives determine the output destinations.
  3. filter directives determine the event processing pipelines. (optional)
  4. system directives set system wide configuration. (optional)
  5. label directives group the output and filter for internal
    routing (optional)
  6. @include directives include other files. (optional)

Fluentd configuration is organized by hierarchy. Each of the directives has different plug-ins and each of the plug-ins has its own parameters. There are numerous plug-ins and parameters associated with each of them. In this post I will go over  on a few of the commonly used ones and focus on giving examples that worked for us here at Coralogix. I also tried to plug pointers to existing documentation to help you locate directive and plug-ins that are not mentioned here. 

Without further ado let’s dive in.

These directives will be present in any Fluentd configuration file:

Source

This is an example of a typical source section in a Fluentd configuration file:

<source>
  @type tail
  path /var/log/msystem.log
  pos_file /var/log/msystem.log.pos
  tag mytag
  <parse>
    @type none
  </parse>
</source>

Let’s examine the different components:

@type tail –  This is one of the most common  Fluentd input plug-ins. There are built-in input plug-ins and many others that are customized. The ‘tail’ plug-in allows Fluentd to read events from the tail of text files. Its behavior is similar to the tail -F command. The file that is read is indicated by ‘path’. Fluentd starts from the last log in the file on restart or from the last position stored in ‘pos_file’, You can also read the file from the beginning by using the ‘read_from_head true’ option in the source directive. When the log file is rotated Fluentd will start from the beginning. Each input plug-in comes with parameters that control its behavior. These are the tail parameters.

Tags allow Fluentd to route logs from specific sources to different outputs based on conditions. E.g – send logs containing the value “compliance” to a long term storage and logs containing the value “stage” to a short term storage.

The parser directive, <parse>, located within the source directive, , opens a format section. This is mandatory. It can use type none (like in our example) if no parsing is needed. This list includes the built-in parsers. There is an option to add a custom parser

One of the commonly used built-in parsers is ‘multiline’. Multiline logs are logs that span across lines. Log shippers will ship these lines as separate logs, making it hard to get the needed information from the log. The ‘Multiline’ parser enables the restructuring and formatting of multiline logs into the one log they represent. You can see a few examples of this parser in action in our examples’ section. See here for a full logging in multiline guide

An example of a multiline configuration file using regex:

<source>
  @type tail
  path /var/log/msystem.log
  pos_file /var/log/msystem.log.pos
  tag mytag
  <parse>
          @type multiline
          # Each firstline starts with a pattern matching the below REGEX.
          format_firstline /^\d{2,4}\-\d{2,4}\-\d{2,4} \d{2,4}\:\d{2,4}\:\d{2,4}\.\d{3,4}/
          format1 /(?<message>.*)/
  </parse>
</source>

These input plug-ins support the parsing directive.

Match  

The Match section uses a rule. Matches each incoming event to the rule and and routes it through an output plug-in. There are different output plug-ins. This is a simple example of a Match section:

<match mytag**>
   @type stdout
</match>

It will match the logs that have a tag name starting with mytag and direct them to stdout. Like the input plug-ins, the output ones come with their own parameters. See The http documentation as an example.  

Combining the two previous directives’ examples will give us a functioning Fluentd configuration file that will read logs from a file and send them to stdout.

Coralogix output plug-in

One of the custom third parties plug-in is Coralogix’s. This is how a match section looks like when logs are shipped to Coralogix. You can read more in our tutorial.

Here is an example:

<match audit>
#Coralogix’s custom third parties plug-in
  @type coralogix
  privatekey "my-private-key"
  appname "mayapp"
  subsystemname "mysub"
#This is a Coralogix specific optional parameter that sets the number of #processes that will work in parallel to fill the output buffer. The value can #be between 1-10 with the default set as 1
  number_of_workers 2
#debug option to enter debug mode while Fluentd is running
  debug true
#When streaming json one can choose which fields to have as output
  log_key_name SOME_KEY_NAME
#Using the timestamp value from the log record
  timestamp_key_name LOG_TIMESTAMP_KEY_NAME
#is_json need to set to true if the output is JSON
  is_json true
</match> 

Filter

Another common directive, but one that is not mandatory, is ‘filter’. The name of this directive is self explanatory. The filter plug-ins allow users to:

  • Filter out events by grepping the value of one or more fields.
  • Enrich events by adding new fields.
  • Delete or mask certain fields for privacy and compliance.
  • Parse text log messages into JSON logs

Filters can be chained together.

As an example, this filter will allow only logs where the key user_name has a value that starts with AR, and continues with consecutive digits to move forward.

<filter>
  @type grep
  <regexp>
   key user_name
   pattern /^AR\d*/
  </regexp>
</filter>

At this point we have enough Fluentd knowledge to start exploring some actual configuration files.

The rest of this document includes more complex examples of Fluentd configurations. They include detailed comments and you can use them as references to get additional information about different plug-ins and parameters or to learn more about Fluentd.

Configuration examples

Example 1

This configuration example shows how to use the rewrite_tag_filter plug-in to separate the logs into two groups and send them with different metadata values.

<source>
#Reading from file located in path and saving the pointer to the file line in pos_file  
  @type tail
  path /var/log/nginx/*.log
  pos_file /var/log/nginx/nginx.log.pos
#tagging the nginx input
  tag nginx
#No parsing is done
  <parse>
    @type none
  </parse>
</source>

<match nginx.**>
#Output for the nginx tagged logs
#This output plugin provides a rule-based mechanism for rewriting tags. Comes with td-agent #but needs to be installed with Fluentd
  @type rewrite_tag_filter
  <rule>
#The field name to which the regular expression is applied
    key message
#Change the tag for logs that include ‘xyz_prod’ in the message field to xyz_prod.nginx. Read more about placeholders here
    pattern /(xyz_prod)/
    tag xyz_prod.${tag}
  </rule>
#This rule will pick all other logs and tag them as other.nginx  
  <rule>
    key message
    pattern /.+/
    tag other.${tag}
  </rule>
</match>

<match other.nginx.**>
#other.nginx logs are sent to Coralogix, using the custom plug-in
  @type coralogix
  privatekey "MyPrivateKey"
  appname "MyAppName1"
  subsystemname "my-other"
#The key to send to Coralogix
  log_key_name message
</match>

<match xyz_prod.nginx.**>
#xyz.nginx logs are sent to Coralogix, using the custom plug-in
  @type coralogix
  privatekey "MyPrivateKey2"
  appname "MyAppName2"
  subsystemname "my-xyz-prod"
#The key to send to Coralogix
  log_key_name message
</match>
#For debug purposes all logs are sent stdout 
<match **>
  @type stdout
</match>

Example 2

In this example a file is read starting with the last line (the default). The message section of the logs is extracted and the multiline logs are parsed into a json format.

<source>
  @type tail
  path M:/var/logs/inputlogs.log
  pos_file M:/var/logs/inputlogs.log.pos
  tag mycompany
  <parse>
# This parse section will find the first line of the log that starts with a date. The log 
#includes  the substring “message”. ‘Parse’ will extract what follows “message” into a #json field called ‘message’. See additional details in #Example 1.
    @type multiline
    format_firstline /^\d{2,4}\-\d{2}\-\d{2,4}/
    format1 /(?<message>.*)/
  </parse>
</source>

#Match is using the Coralogix plug-in 
<match mycompany.**>
#Coralogix’s custom third parties plug-in
  @type coralogix
  privatekey **see instructions below**
  appname **see instructions below **
  subsystemname **see instructions below**
#This parameter will extract and send only the field ‘message’ to Coralogix
  log_key_name message  
#In case the input to <match> is a json object, set to true 
  is_json false
</match>

Example 3

This example uses the http input plug-in. This plug-in enables you to gather logs while sending them to an end point. It uses the enable_ruby option to transform the logs and uses the copy plug-in to send logs to two different inputs.

<source>
  @type http
  port 1977
  bind 0.0.0.0
  tag monitor
#This indicates the max size of a posted object   
  body_size_limit 2m
#This is the timeout to keep a connection alive
  keepalive_timeout 20s
#If true adds the http prefix to the log
  add_http_headers true
#If true adds the remote, client address, to the log. If there are multiple forward headers in the request it will take the first one  
  add_remote_addr true
  <parse> 
     @type none
  </parse>
</source>

<filter monitor>
#record_transformer is a filter plug-in that allows transforming, deleting, and adding events
  @type record_transformer
#With the enable_ruby option, an arbitrary Ruby expression can be used inside #${...}
  enable_ruby
#Parameters inside <record> directives are considered to be new key-value pairs
  <record>
#Parse json data inside of the nested field called “log”. Very useful with #escaped fields. If log doesn’t exist or is not a valid json, it will do #nothing. 
    log ${JSON.parse(record["log"]) rescue record["log"]}
#This will create a new message field under root that includes a parsed log.message field.
    message ${JSON.parse(record.dig(“log”, “message”)) rescue ""}
  </record>
</filter>
<match monitor>
  @type copy
  <store>
#Coralogix’s custom third parties plug-in
    @type coralogix
    privatekey "YOUR_PRIVATE_KEY"
    appname "$message.reqHost"
    subsystemname "Cloud Monitor"
    log_key_name message
    is_json true
  </store>

  <store>
    @type stdout
    output_type json
  </store>
</match>

<match **>
#Split is a third party output plug-in. It helps split a log and parse #specific fields.
  @type split
#separator between split elements 
  separator   \s+
#regex that matches the keys and values within the splitted key
  format      ^(?<key>[^=]+?)=(?<value>.*)$
#Key that holds the information to be splitted
  key_name message
#keep the original field
  reserve_msg yes
  keep_keys HTTP_PRIVATE_KEY
</match>

Example 4

This is another example of fixing multiline logs. It uses the multi_format plug-in to deal with different pattern multiline logs.

<source>
#Reading from a file indicated by path. Pos_file holds the pointer to the last line read.  
  @type tail
  path /var/log/haproxy.log
  pos_file /var/log/haproxy.log.pos
  encoding UTF-8
#The interval of refreshing the watch file list. Default is 60 seconds.
  refresh_interval 5s
  tag myproxy
  <parse>
#This parse section uses the multi_format plug-in. This plug-in needs to be #downloaded and doesn’t come with Fluentd. After installing it you will be able #to Use multiple <pattern>s to specify multiple parser formats. In this #configuration file we have 2 patterns being formatted. 
    @type multi_format
    <pattern>
      format /^(?<date>[a-zA-Z]{3}\s*\d+ \d{2}\:\d{2}\:\d{2}) (?<hostname>.+?) (?<process_name>.*)\[(?<pid>\d+)\]\: (?<client_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\:(?<client_port>\d+) \[(?<accept_date>[^\]]*)\] (?<frontend_name>.+?) (?<backend_name>.+?)\/(?<server_name>.+?) (?<Tq>[0-9\-]+)\/(?<Tw>[0-9\-]+)\/(?<Tc>[0-9\-]+)\/(?<Tr>[0-9\-]+)\/(?<Tt>[0-9\-]+) (?<status_code>\d+) (?<bytes_read>\d+) (?<captured_request_cookie>.+?) (?<captured_response_cookie>.+?) (?<termination_state>.+?) (?<actconn>\d+)\/(?<feconn>\d+)\/(?<beconn>\d+)\/(?<srv_conn>\d+)\/(?<retries>\d+) (?<srv_queue>\d+)\/(?<backend_queue>\d+) \{(?<captured_request_headers>.*?)\} \"(?<http_request>.*)\"$/
#Specifies time field for event time. If the event doesn't have this field, #current time is used.
      time_key date
#Processes value using specific formats
      time_format %b  %e %T
#When true keeps the time key in the log
      keep_time_key true
    </pattern>
    <pattern>
      format /^(?<date>[a-zA-Z]{3}\s*\d+ \d{2}\:\d{2}\:\d{2}) (?<hostname>.+?) (?<process_name>.*)\[(?<pid>\d+)\]\: (?<message>.+)$/
#Specifies time field for event time. If the event doesn't have this field, #current time is used.
      time_key date
#Processes value using specific formats
      time_format %b  %e %T
#When true keeps the time key in the log
      keep_time_key true
    </pattern>
  </parse>
</source>

<filter haproxy.**>
#record_transformet is a filter plug-in that allows transforming, deleting, and #adding events
  @type record_transformer
#With the enable_ruby option, an arbitrary Ruby expression can be used inside #${...}
  Enable_ruby
#Parameters inside <record> directives are considered to be new key-value pairs
  <record>
    duration ${record["Tt"].to_i}
    headers ${record.fetch("captured_request_headers", "|").split('|')}
  </record>
</filter>
 
<filter haproxy.**>
#record_transformet is a filter plug-in that allows transforming, deleting, and #adding events
  @type record_transformer
#With the enable_ruby option, an arbitrary Ruby expression can be used inside #${...}
  Enable_ruby
#Parameters inside <record> directives are considered to be new key-value pairs
  <record>
    x-forward-for ${record["headers"].fetch(0, "").split(',').fetch(0, "")}
    x-redmine-api-key ${record.fetch("http_request", "").scan(/(?:^|[?&])key=([^& ]*)/).fetch(0, []).fetch(0, record["headers"].fetch(1, ""))}
  </record>
# Gets a list of keys and deletes them
  remove_keys ["headers"]
</filter>

<match haproxy.**>
#Coralogix’s custom third parties plug-in
  @type coralogix
  privatekey "MyPrivateKey"
  appname "MyAppName"
  subsystemname "MySub"
#When sending a json formatted log
  is_json true
</match>

Example 5

This example uses label in order to route the logs through its Fluentd journey. At the end we send Fluentd logs to stdout for debug purpose.

# Read Tomcat logs
<source>
#Logs are read from a file located at path. A pointer to the last position in #the log file is located at pos_file
  @type tail
#Labels allow users to separate data pipelines. You will see the lable section #down the file.
  @label @AGGREGATION
  path /usr/local/tomcat/logs/*.*
#adds the watched file path as a value of a new key filename in the log
  path_key filename
#This path will not be watched by fluentd
  exclude_path ["/usr/local/tomcat/logs/gc*"]
  pos_file /fluentd/log/tomcat.log.pos
  <parse>
#This parse section uses the multi_format plug-in. This plug-in needs to be #downloaded and doesn’t come with Fluentd. After installing it users can #configure multiple <pattern>s to #specify multiple parser formats. In this #configuration file we have 2 patterns being formatted. 
    @type multi_format
    # Access logs
    <pattern>
      format regexp
      expression /^(?<client_ip>.*?) \[(?<timestamp>\d{2}\/[a-zA-Z]{3}\/\d{4}\:\d{2}\:\d{2}\:\d{2} \+\d{4})\] \"(?<message>.*?)\" (?<response_code>\d+) (?<bytes_send>[0-9\-]+) (?<request_time>\d+)$/
      types response_code:integer,bytes_send:integer,request_time:integer
#Specifies time field for event time. If the event doesn't have this field, #current time is used.
      time_key timestamp
#Processes value using specific formats
      time_format %d/%b/%Y:%T %z
#When true keeps the time key in the log
      keep_time_key true
    </pattern>
# Stacktrace or undefined logs are kept as is
    <pattern>
      format none
    </pattern>
  </parse>
  tag tomcat
</source>

# Read Cloud logs
<source>
  @type tail
  @label @AWS
  path /usr/local/tomcat/logs/gc*
  pos_file /fluentd/log/gc.log.pos
  format none
  tag gclogs
</source>

# Route logs according to their types
<label @AGGREGATION>
  # Strip log message using a filter section
  <filter tomcat.**>
#record_transformer is a filter plug-in that allows transforming, deleting, and #adding events
    @type record_transformer
#With the enable_ruby option, an arbitrary Ruby expression can be used inside #${...}
    enable_ruby
#Parameters inside <record> directives are considered to be new key-value pairs
    <record>
      message ${record["message"].strip rescue record["message"]}
    </record>
  </filter>
# Delete undefined character
# This filter section filters the tomcat logs 
  <filter tomcat.**>
#record_transformer is a filter plug-in that allows transforming, deleting, and #adding events
    @type record_transformer
    Enable_ruby
#Parameters inside <record> directives are considered to be new key-value pairs
    <record>
      message ${record["message"].encode('UTF-8', invalid: :replace, undef: :replace, replace: '?') rescue record["message"]}
    </record>
  </filter>

# Concat stacktrace logs to processing
  <filter tomcat.**>
#A plug-in that needs to be installed
    @type concat
#This is the key for part of the multiline log
    key message
#This defines the separator
    separator " "
#The key to determine which stream an event belongs to
    stream_identity_key filename
#The regexp to match beginning of multiline.
    multiline_start_regexp /^((\d{2}\/\d{2}\/\d{4} \d{2}\:\d{2}\:\d{2}\.\d{1,3})|(\d{2}\-[a-zA-Z]{3}\-\d{4} \d{2}\:\d{2}\:\d{2}\.\d{1,3})|NOTE\:) /
#Use timestamp of first record when buffer is flushed.
    use_first_timestamp true
#The number of seconds after which the last received event log will be flushed
    flush_interval 5
#The label name to handle events caused by timeout
    timeout_label @PROCESSING
  </filter>
# Route logs to processing
  <match tomcat.**>
    @type relabel
    @label @PROCESSING
  </match>
</label>

 # Process simple logs before sending to Coralogix
<label @PROCESSING>
# Parse stacktrace logs
  <filter tomcat.**>
    @type parser
    format /^(?<timestamp>\d{2}\/\d{2}\/\d{4} \d{2}\:\d{2}\:\d{2}\.\d{1,3}) (?<hostname>.*?)\|(?<thread>.*?)\|(?<severity>[A-Z ]+)\|(?<TenantId>.*?)\|(?<UserId>.*?)\|(?<executor>.*?) \- (?<message>.*)$/
#Specifies time field for event time. If the event doesn't have this field, #current time is used.
    time_key timestamp
#Processes value using specific formats
    time_format %d/%m/%Y %T.%L
#When true keeps the time key in the log
    keep_time_key true
    key_name message
    reserve_data true
    emit_invalid_record_to_error false
  </filter>

 # Parse access logs request info
  <filter tomcat.**>
    @type parser
    format /^(?<method>GET|HEAD|POST|PUT|DELETE|CONNECT|OPTIONS|TRACE|PATCH) (?<path>\/.*?) (?<protocol>.+)$/
    key_name message
    hash_value_field request
    reserve_data true
    emit_invalid_record_to_error false
  </filter>

# Other logs  
  <filter tomcat.**>
#This is a filter plug-in that pases the logs
    @type parser
    format /^(?<timestamp>\d{2}\-[a-zA-Z]{3}\-\d{4} \d{2}\:\d{2}\:\d{2}\.\d{1,3}) (?<severity>[A-Z ]+) \[(?<thread>.*?)\] (?<executor>.*?) (?<message>.*)$/
#Specifies time field for event time. If the event doesn't have this field, #current time is used.
    time_key timestamp
#Processes value using specific formats
    time_format %d-%b-%Y %T.%L
#When true keeps the time key in the log
    keep_time_key true
#message is a key in the log to be filtered
    key_name message
#keep the other key:value pairs
    reserve_data true
#Emit invalid records to @ERROR label. Invalid cases are key not exist, format #is not matched, and unexpected error. If you want to ignore these errors set #to false.
    emit_invalid_record_to_error false
  </filter>

# Add filename and severity
  <filter tomcat.**>
#record_transformet is a filter plug-in that allows transforming, deleting, and #adding events
    @type record_transformer
#With the enable_ruby option, an arbitrary Ruby expression can be used inside #${...}
    Enable_ruby
#Parameters inside <record> directives are considered to be new key-value pairs
    <record>
#Overwrite the value of filename with the filename itself without the full path
      filename ${File.basename(record["filename"])}
#Takes the value of the field “severity” and trims it. If there is no severity #field create it and return the value DEBUG
      severity ${record["severity"].strip rescue "DEBUG"}
    </record>
  </filter> 
  # Route logs to output
  <match tomcat.**>
#Routes all logs to @labe
    @type relabel
    @label @CORALOGIX
  </match>
</label>

# Send logs to Coralogix
<label @CORALOGIX>
  <match tomcat.**>
#Coralogix’s custom third parties plug-in
    @type coralogix
#3 mandatory parameters
    privatekey "#{ENV['PRIVATE_KEY']}"
    appname "#{ENV['APP_NAME']}"
    subsystemname "#{ENV['SUBSYSTEM_NAME']}"
#set to true when json logs are sent
    is_json true
  </match>
</label>

# Send GC logs to AWS S3 bucket
#<label> indicates that only AWS labeled logs will be processed here. They will #skip sections not labeled with AWS
<label @AWS>
  <match gclogs.**>
#The S3 output plugin is included with td-agent (not with Fluentd). You can get #information about its specific parameters in the documentation
   @type s3
   aws_key_id "#{ENV['AWS_ACCESS_KEY_ID']}"
   aws_sec_key "#{ENV['AWS_SECRET_ACCESS_KEY']}"
   s3_bucket "#{ENV['S3Bucket']}"
   s3_region us-east-1
   path /logs/"#{ENV['SUBSYSTEM_NAME']}"
   buffer_path /fluentd/logs
   store_as text
   utc
   time_slice_format %Y%m%d%H
   time_slice_wait 10m
   s3_object_key_format "%{path}%{time_slice}_#{Socket.gethostname}%{index}.%{file_extension}"
   buffer_chunk_limit 256m
   <buffer time>
    timekey      1h # chunks per hours ("3600" also available)
    timekey_wait 5m # 5mins delay for flush ("300" also available)
   </buffer>
  </match>
</label>
 
# Print internal Fluentd logs to console for debug
<match fluent.**>
  @type stdout
  output_type hash
</match>

Example 6

This example uses a third party @split output plugin, to split a log based on some rule (in this case newline) into different logs.

<source>
#This plug-in enables gathering logs from an end point
  @type http
#These parameters indicate the port and address to listen to
  port 1977
  bind 0.0.0.0
#The size limit of the posted logs
  body_size_limit 2m
#The timeout limit for keeping the connection alive
  keepalive_timeout 20s
#Add http prefix headers to the log
  add_http_headers true
#Add remote_addr field to the log. The value of remote_addr is the client's #address.
  add_remote_addr true
#This is a deprecated parameter. We kept it here, so that you will be aware of #it. Newer config’s will have a parse section instead.
  format none
</source>

#Apply the filter to cloudmonitor tagged logs
<filter cloudmonitor>
#record_transformer is a filter plug-in that allows transforming, deleting, and #adding events
  @type record_transformer
#With the enable_ruby option, an arbitrary Ruby expression can be used inside #${...}
  Enable_ruby
#Parameters inside <record> directives are considered to be new key-value pairs
  <record>
    log ${JSON.parse(record["log"]) rescue record["log"]}
    message ${JSON.parse(record["log"]).fetch("message", "") rescue ""}
  </record>
</filter>

<match cloudmonitor>
#Copy is an output plugin that copies events to multiple outputs.
  @type copy
#Specifies the storage destinations. 
  <store>
#Coralogix’s custom third parties plug-in
    @type coralogix
    privatekey $HTTP_PRIVATE_KEY
    appname "$message.reqHost"
    subsystemname "Cloud Monitor"
#This parameter will extract and send only the field ‘message’ to Coralogix
    log_key_name message
#In case the input to <match> is a json object, set to true
    is_json true
  </store>
  <store>
#This a second output stream that goes to stdout
    @type stdout
    output_type json
  </store>
</match>

<match **>
#Split is a third party output plug-in. It helps split a log and parse specific #fields. In this case we  take the cloudmonior tagged logs, identify logs with #log field that has newlines in it, and split these lines into different logs #with the fied message, holding the log data.
  @type split
  output_tag cloudmonitor
  output_key log
  format newline
  key_name message
#keep this key during the transformation
  keep_keys HTTP_PRIVATE_KEY
</match>

Example 7

The following configuration reads the input files starting with the first line. Then transform multi line logs into a json format. It sends the logs to stdout.

<source>
  @type tail
#The parameter ‘read_from_head’ is set to true. It will read all files from the #first line instead of #the default last line
  tag audit
  read_from_head true
# Adds the file name to the sent log.  
  path_key filename
  path /etc/logs/product_failure/*.*.testInfo/smoke*-vsim-*/mroot/etc/log/auditlog.log.*
  pos_file /etc/logs/product_failure/audit_logs.fluentd.pos
  <parse>
#The multiline parser plugin parses multiline logs. This plugin is multiline #version of regexp parser.
    @type multiline
#This will match the first line of the log to be parsed. The plugin can skip #the logs until format_firstline is matched.
    format_firstline /^(?<timestamp>[a-zA-Z]{3} [a-zA-Z]{3} *[0-9]{1,2} [0-9]{2}:[0-9]{2}:[0-9]{2}(?: [A-Z]+)?)/
#Specifies regexp patterns. For readability, you can separate regexp patterns #into multiple format1 … formatN. The regex will match the log and each named #group will become a key:value pair in a json formatted log. The key will be #the group name and the value will be the group value.
    format1 /^(?<timestamp>[a-zA-Z]{3} [a-zA-Z]{3} *[0-9]{1,2} [0-9]{2}:[0-9]{2}:[0-9]{2}(?: [A-Z]+)?) \[(?<machine>[^:]+):(?<shell>[^:]+):(?<severity>[^\]]+)\]: (?<message>.*)/
  </parse>
</source>

# Send to STDOUT
<match *>
  @type stdout
</match>


Start solving your production issues faster

Let's talk about how Coralogix can help you better understand your logs

Managed, Scaled and Compliant ELK Stack

No credit card required

Get a personalized demo

Jump on a call with one of our experts and get a live personalized demonstration