Filebeat

Coralogix provides seamless integration with Filebeat so you can send your logs from anywhere and parse them according to your needs.

Supported versions

Coralogix supports these versions of Filebeat:

• Filebeat 7.x

• Filebeat 8.x

Note! To avoid breaking changes between these major versions, you should not upgrade directly from v7 to v8.

Prerequisites

• Filebeat installed

• Your domain-dependent URL, selected from the table below

• Our SSL certificate installed on your system, presented in the table below.

Notes:

The Logstash endpoint is supported only for the following Coralogix domains: EU1, US1, and AP1. Other Coralogix domains do not support the Logstash endpoint.

For assistance with Filebeat or other configurations in unsupported domains, please contact Coralogix Support.

General

Private Key - A unique ID that represents your company. The Send-Your-Data API key can be found under ‘settings’->’ send your logs’. It is located in the upper left corner.

Application Name - The name of your main application, for example, a company named "SuperData" would probably insert the "SuperData" string parameter or if they want to debug their test environment they might insert the "SuperData– Test".

SubSystem Name - Your application probably has multiple subsystems, for example, Backend servers, Middleware, Frontend servers, etc. in order to help you examine the data you need, inserting the subsystem parameter is vital.

Installation

For a quick setup of Filebeat on your server, you can use prepared scripts.

Go to the folder with your Filebeat configuration file (filebeat.yml) and execute (as root):

deb

curl -sSL https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/scripts/install-deb.sh | bash

rpm

curl -sSL https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/scripts/install-rpm.sh | bash

This script will install Filebeat on your machine, prepare configuration and download Coralogix SSL certificates.

Note: If you want to install a specific version of Filebeat you should pass version number with environment variable before script run:

export FILEBEAT_VERSION=7.17.8

Configuration

Open your Filebeat configuration file and configure it to use Logstash (Make sure you disable Elasticsearchoutput). For more information about configuring Filebeat to use Logstash please refer to https://www.elastic.co/guide/en/beats/filebeat/current/config-filebeat-logstash.html

Point your Filebeat to output to Coralogix Logstash server (replace the Logstash Server URL with the corresponding entry from the table above):

logstashserver.Cluster URL:5044

or if you want to use an encrypted connection (recommended):

logstashserver.Cluster URL:5015

Here is a basic example of filebeat.yml:

#============================== Filebeat Inputs ===============================

filebeat.inputs:
- type: log
  paths:
  - "/var/log/your_app/your_app.log"

fields_under_root: true
fields:
  PRIVATE_KEY: "YOUR_PRIVATE_KEY"
  COMPANY_ID: YOUR_COMPANY_ID
  APP_NAME: "APP_NAME"
  SUB_SYSTEM: "SUB_NAME"

#----------------------------- Logstash output --------------------------------

output.logstash:
  enabled: true
  hosts: ["logstashserver.Cluster URL:5015"]
  ttl: 60s
  tls.certificate_authorities: ["<path to folder with certificates>/ca.crt"]
  ssl.certificate_authorities: ["<path to folder with certificates>/ca.crt"]

Note: If you want to send all additional metadata, the fields_under_root option should be equals to true.

If you have multiline logs like:

2019-08-31 14:27:33 [main] ERROR Main - Exception
javax.management.RuntimeErrorException: null
    at Main.main(Main.java:16) ~[bin/:na]

You can use multiline pattern:

filebeat.inputs:
- type: log
  paths:
  - "/var/log/your_app/your_app.log"
  multiline:
    pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \['
    negate: true
    match: after

The pattern should match the beginning of each record.

Docker

Build a Docker image with your filebeat.yml:

FROM docker.elastic.co/beats/filebeat:7.17.8

LABEL description="Filebeat logs watcher"

# Adding configuration file and SSL certificates for Filebeat
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
COPY ca.crt /etc/ssl/certs/Coralogix.crt

# Changing permission of configuration file
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml

# Return to deploy user
USER filebeat

Before deploying your container don't forget to mount volume with your logs.

URLs and certificates

Validation

Test Filebeat by running it and monitoring the logs.

1. Modify the user credentials in filebeat.yml and specify a user who is authorized to publish events.

sudo chown root filebeat.yml 

1. By default, Filebeat sends all of its output to Syslog. When you run Filebeat in the foreground, you can use the -e command line flag to redirect the output to standard error instead, as in the example below.

sudo ./filebeat -e

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at support@coralogix.com.

Previous Logstash

Next Fluentd

Theme

Light

light

dark