This tutorial provides a step-by-step guide on to how install and configure Filebeat to send logs from a file to your Coralogix team over TLS. It does this using a deployment of Filebeat on a single Amazon Linux 2 instance.
Note! Filebeat can be used to ship logs from a variety of sources, including Syslog, Docker, and Windows Environments.
Learn how to:
Coralogix supports these versions of Filebeat:
Note! To avoid breaking changes between these major versions, do not upgrade directly from v7 to v8.
This section demonstrates how to deploy Filebeat on a single Amazon Linux 2 instance. General instructions for installing and configuring Filebeat and sending your data to Coralogix can be found here.
Install and configure Filebeat v7.17 on your Linux distribution.
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.8-linux-x86_64.tar.gz tar xzvf filebeat-7.17.8-linux-x86_64.tar.gz cd filebeat-7.17.8-linux-x86_64/
To configure Filebeat, modify the main parts of the configuration file filebeat.yml
: modules, inputs, fields, outputs.
Configure the modules. These are Filebeat inputs enabling the input and parser.
The example below configures the Fortinet / Firewall module, enabling Filebeat to ingest Syslog data from FortiGate Firewall on port 9004/UDP and parse Syslog messages in JSON format.
#========================== Modules configuration ============================= filebeat.modules: - module: fortinet firewall: enabled: true var.input: udp var.syslog_host: 0.0.0.0 var.syslog_port: 9004
Note! Modules change dramatically between different versions of Filebeat. Previous versions of Filebeat do not have all modules available.
To configure Filebeat manually (rather than using modules), specify a list of inputs in the filebeat.inputs
section of the filebeat.yml
. Inputs specify how Filebeat locates and processes input data.
The log input in the example below enables Filebeat to ingest data from the log file. It then points Filebeat to the logs folder and uses a wildcard *.log
to collect all files ending with .log
.
#=========================== Filebeat inputs ============================= #------------------------------ Log input -------------------------------- - type: log # Change to true to enable this input configuration. enabled: false # Paths that should be crawled and fetched. Glob based paths. # To fetch all ".log" files from a specific level of subdirectories # /var/log/*/*.log can be used. # For each file found under this path, a harvester is started. # Make sure not file is defined twice as this can lead to unexpected behaviour. paths: - /var/log/*.log #- c:\programdata\elasticsearch\logs\*
Apply additional configuration settings (such as fields
, include_lines
, exclude_lines
, multiline
) to the lines harvested from logs. The options that you specify are applied to all of the files harvested by a single input.
To apply different configuration settings to different files, define multiple input sections.
Note! Ensure a file is not defined more than once across all inputs because this can lead to unexpected behavior.
filebeat.inputs: - type: log paths: - /var/log/*.log fields: PRIVATE_KEY: '<coralogix_send-your-data-api-key>' COMPANY_ID: <companyID> APP_NAME: '<application_name>' SUB_SYSTEM: '<subsystem_name>' fields_under_root: true
1. Configure Filebeat to write specific outputs by setting options in the output
section of the configuration file.
The logstash
output in the example below enables Filebeat to ship data to Logstash. It points Filebeat to the Coralogix logstash in the coralogix.com
domain and points Filebeat to the TLS and SSL certificates (same certificate) that are required to ship data securely to Coralogix.
Note! Only a single output may be defined.
# ================================= Logstash output ============================= output.logstash: enabled: true hosts: ['logstashserver.coralogix.com:5015'] tls.certificate_authorities: ['/usr/share/Coralogix-EU.crt'] ssl.certificate_authorities: ['/usr/share/Coralogix-EU.crt']
2. Download and store the certificate in a location accessible by Filebeat.
filebeat.yml: |- # ============================== Filebeat Inputs =============================== #-------------------------------- logs input --------------------------------- filebeat.inputs: - type: log paths: - "/var/log/your_app/your_app.log" line_delimiter: "\n" max_message_size: 10MiB timeout: 300s enable_metric: true #------------------------ Coralogix fields configuration -------------------- fields: PRIVATE_KEY: '<coralogix_privatekey>' COMPANY_ID: <companyID> APP_NAME: '<application_name>' SUB_SYSTEM: '<subsystem_name>' fields_under_root: true #========================== Modules configuration ============================= Filebeat.modules: - module: fortinet Firewall: enabled: true var.input: udp var.syslog_host: 0.0.0.0 var.syslog_port: 9004 fields: PRIVATE_KEY: '<coralogix_send-your-data-api-key>' COMPANY_ID: <companyID> APP_NAME: '<application_name>' SUB_SYSTEM: '<subsystem_name>' fields_under_root: true # ================================= Logstash output ============================= #------------------------- Coralogix Logstash output —----------------------- output.logstash: enabled: true hosts: ['logstashserver.coralogix.com:5015'] tls.certificate_authorities: ['/usr/share/Coralogix-EU.crt'] ssl.certificate_authorities: ['/usr/share/Coralogix-EU.crt']
EU | IN | US | |
Cluster Domain | coralogix.com | app.coralogix.in | coralogix.us |
SSL Certificates | https://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/Coralogix-EU.crt | https://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/Coralogix-IN.pem | https://www.amazontrust.com/repository/AmazonRootCA1.pem |
Logstash Server URL | logstashserver.coralogix.com | logstash.app.coralogix.in | logstashserver.coralogix.us |
Test Filebeat by running it and monitoring the logs.
1. Modify the user credentials in filebeat.yml
and specify a user who is authorized to publish events.
sudo chown root filebeat.yml
2. By default, Filebeat sends all of its output to Syslog. When you run Filebeat in the foreground, you can use the -e
command line flag to redirect the output to standard error instead, as in the example below.
sudo ./filebeat -e
To increase the verbosity of debug messages, use the -d
command line flag to debug selectors.
./filebeat -e -d "*"
You may encounter certain common messages as follows:
INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s
2022-12-19T19:35:41.758Z INFO [publisher_pipeline_output] pipeline/output.go:101 Connecting to backoff(async(tcp://logstashserver.coralogix.com:5015)) 2022-12-19T19:35:41.886Z INFO [publisher_pipeline_output] pipeline/output.go:111 Connection to backoff(async(tcp://logstashserver.coralogix.com:5015)) established
Need help?
Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.
Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].