Open in ChatGPTOpen in Claude

Filebeat

Note

This integration is maintained for legacy support and is not recommended for new use cases. We recommend using OpenTelemetry (OTel) for a more modern, flexible, and industry-standard observability solution.

Coralogix integrates with Filebeat so you can send your logs from anywhere and parse them according to your needs.

Supported versions

Coralogix supports these versions of Filebeat:

• Filebeat 7.x

• Filebeat 8.x

To avoid breaking changes between these major versions, do not upgrade directly from v7 to v8.

What you need

• Filebeat installed

• A Logstash instance that you run in your own infrastructure, configured to receive Beats input and forward it to Coralogix. Coralogix no longer hosts a Logstash server for Beats shippers. See Run your own Logstash instance for the Logstash input, filter, and output configuration.

General

Private Key: A unique ID that represents your company. The Send-Your-Data API key can be found under ‘settings’->’ send your logs’. It is located in the upper left corner.

Application Name: The name of your main application, for example, a company named "SuperData" would probably insert the "SuperData" string parameter or if they want to debug their test environment they might insert the "SuperData - Test".

SubSystem Name: Your application probably has multiple subsystems, for example, Backend servers, Middleware, Frontend servers, etc. in order to help you examine the data you need, inserting the subsystem parameter is vital.

Installation

For a quick setup of Filebeat on your server, you can use prepared scripts.

Go to the folder with your Filebeat configuration file (filebeat.yml) and execute as root:

deb

curl -sSL https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/scripts/install-deb.sh | bash

rpm

curl -sSL https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/scripts/install-rpm.sh | bash

This script installs Filebeat on your machine, copies the filebeat.yml from your current directory to /etc/filebeat/, and starts the Filebeat service. It does not write the configuration for you, so set the output.logstash host in your filebeat.yml before you run it, as shown below.

The script also downloads a Coralogix SSL certificate. That step is a leftover from the retired hosted Logstash service. A self-managed Logstash instance does not use it, and the downloaded file can be ignored.

To install a specific version of Filebeat, pass the version number in an environment variable before you run the script:

export FILEBEAT_VERSION=7.17.8

Configuration

Open your Filebeat configuration file and configure it to use Logstash, making sure you disable the Elasticsearch output. For more information about configuring Filebeat to use Logstash, see the Elastic documentation.

Point your Filebeat to output to your own Logstash instance:

<logstash-ip-address>:5044

Note

The Coralogix-hosted Logstash endpoints (logstashserver.coralogix.com, logstash.coralogix.in, logstashserver.coralogix.us) are retired and no longer accept data. See Deprecation of Coralogix-hosted Logstash.

Set the General values (Send-Your-Data API key, application name, and subsystem name) in your Logstash configuration, not in filebeat.yml.

Here is a basic example of filebeat.yml:

#============================== Filebeat Inputs ===============================

filebeat.inputs:
- type: log
  paths:
  - "/var/log/your_app/your_app.log"

#----------------------------- Logstash output --------------------------------

output.logstash:
  enabled: true
  hosts: ["<logstash-ip-address>:5044"]

If you have multiline logs like:

2019-08-31 14:27:33 [main] ERROR Main - Exception
javax.management.RuntimeErrorException: null
    at Main.main(Main.java:16) ~[bin/:na]

You can use multiline pattern:

filebeat.inputs:
- type: log
  paths:
  - "/var/log/your_app/your_app.log"
  multiline:
    pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \['
    negate: true
    match: after

The pattern should match the beginning of each record.

Docker

Build a Docker image with your filebeat.yml:

FROM docker.elastic.co/beats/filebeat:7.17.8

LABEL description="Filebeat logs watcher"

# Adding configuration file for Filebeat
COPY filebeat.yml /usr/share/filebeat/filebeat.yml

# Changing permission of configuration file
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml

# Return to deploy user
USER filebeat

Before deploying your container, mount the volume with your logs.

Validation

Test Filebeat by running it and monitoring the logs.

1. Modify the user credentials in filebeat.yml and specify a user who is authorized to publish events.

sudo chown root filebeat.yml 

2. By default, Filebeat sends all of its output to Syslog. When you run Filebeat in the foreground, you can use the -e command line flag to redirect the output to standard error instead, as in the example below.

sudo ./filebeat -e