# Filebeat

Note

This integration is maintained for legacy support and is not recommended for new use cases. We recommend using [OpenTelemetry (OTel)](https://coralogix.com/docs/opentelemetry/getting-started/index.md) for a more modern, flexible, and industry-standard observability solution.

Coralogix provides seamless integration with Filebeat so you can send your logs from anywhere and parse them according to your needs.

## Supported versions

Coralogix supports these versions of Filebeat:

- Filebeat 7.x
- Filebeat 8.x

**Note!** To avoid breaking changes between these major versions, you should **not** upgrade directly from v7 to v8.

## Prerequisites

- Filebeat [installed](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html)
- Your [domain](https://coralogix.com/docs/user-guides/account-management/account-settings/coralogix-domain/index.md)-dependent URL, selected from the table below
- Our SSL certificate installed on your system, presented in the table below.

| Coralogix Region | EU                                                                               | IN                                                                               | US                                                       |
| ---------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------- |
| Coralogix Domain | coralogix.com                                                                    | coralogix.in                                                                     | coralogix.us                                             |
| SSL Certificates | https://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/Coralogix-EU.crt | https://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/Coralogix-IN.pem | https://www.amazontrust.com/repository/AmazonRootCA1.pem |
| Logstash Server  | logstashserver.coralogix.com                                                     | logstash.coralogix.in                                                            | logstashserver.coralogix.us                              |

**Notes**:

The Logstash endpoint is supported only for the following Coralogix domains: **EU1**, **US1**, and **AP1**. Other Coralogix domains do not support the Logstash endpoint.

For assistance with Filebeat or other configurations in unsupported domains, please contact **Coralogix Support**.

## General

**Private Key** - A unique ID that represents your company. The [Send-Your-Data API key](https://coralogix.com/docs/user-guides/account-management/api-keys/send-your-data-api-key/index.md) can be found under ‘settings’->’ send your logs’. It is located in the upper left corner.

**Application Name** - The name of your main application, for example, a company named "SuperData" would probably insert the "SuperData" string parameter or if they want to debug their test environment they might insert the "SuperData– Test".

**SubSystem Name** - Your application probably has multiple subsystems, for example, Backend servers, Middleware, Frontend servers, etc. in order to help you examine the data you need, inserting the subsystem parameter is vital.

## Installation

For a quick setup of `Filebeat` on your server, you can use prepared [scripts](https://github.com/coralogix/integrations-docs/tree/master/integrations/filebeat/scripts).

Go to the folder with your `Filebeat` configuration file **(filebeat.yml)** and execute **(as root)**:

### deb

```bash
curl -sSL https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/scripts/install-deb.sh | bash
```

### rpm

```bash
curl -sSL https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/scripts/install-rpm.sh | bash
```

This script will install `Filebeat` on your machine, prepare configuration and download *Coralogix* SSL certificates.

**Note:** If you want to install a specific version of `Filebeat` you should pass version number with environment variable before script run:

```bash
export FILEBEAT_VERSION=7.17.8
```

## Configuration

Open your `Filebeat` configuration file and configure it to use `Logstash` (Make sure you disable `Elasticsearch`output). For more information about configuring `Filebeat` to use `Logstash` please refer to [https://www.elastic.co/guide/en/beats/filebeat/current/config-filebeat-logstash.html](https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html)

Point your `Filebeat` to output to *Coralogix* Logstash server (replace the Logstash Server URL with the corresponding entry from the table above):

```text
logstashserver.<coralogix_domain>:5044
```

or if you want to use an encrypted connection **(recommended)**:

```text
logstashserver.<coralogix_domain>:5015
```

Here is a basic example of **filebeat.yml**:

```yaml
#============================== Filebeat Inputs ===============================

filebeat.inputs:
- type: log
  paths:
  - "/var/log/your_app/your_app.log"

fields_under_root: true
fields:
  PRIVATE_KEY: "YOUR_PRIVATE_KEY"
  COMPANY_ID: YOUR_COMPANY_ID
  APP_NAME: "APP_NAME"
  SUB_SYSTEM: "SUB_NAME"

#----------------------------- Logstash output --------------------------------

output.logstash:
  enabled: true
  hosts: ["logstashserver.<coralogix_domain>:5015"]
  ttl: 60s
  tls.certificate_authorities: ["<path to folder with certificates>/ca.crt"]
  ssl.certificate_authorities: ["<path to folder with certificates>/ca.crt"]
```

**Note:** If you want to send all additional metadata, the **fields_under_root** option should be equals to *true*.

If you have *multiline* logs like:

```text
2019-08-31 14:27:33 [main] ERROR Main - Exception
javax.management.RuntimeErrorException: null
    at Main.main(Main.java:16) ~[bin/:na]
```

You can use multiline pattern:

```yaml
filebeat.inputs:
- type: log
  paths:
  - "/var/log/your_app/your_app.log"
  multiline:
    pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \['
    negate: true
    match: after
```

The pattern should match the beginning of each record.

### Docker

Build a Docker image with your **filebeat.yml**:

```dockerfile
FROM docker.elastic.co/beats/filebeat:7.17.8

LABEL description="Filebeat logs watcher"

# Adding configuration file and SSL certificates for Filebeat
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
COPY ca.crt /etc/ssl/certs/Coralogix.crt

# Changing permission of configuration file
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml

# Return to deploy user
USER filebeat
```

Before deploying your container **don't forget** to mount volume with your logs.

## URLs and certificates

|                     | EU                                                                               | IN                                                                               | US                                                       |
| ------------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------- |
| Domain              | coralogix.com                                                                    | app.coralogix.in                                                                 | coralogix.us                                             |
| SSL Certificates    | https://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/Coralogix-EU.crt | https://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/Coralogix-IN.pem | https://www.amazontrust.com/repository/AmazonRootCA1.pem |
| Logstash Server URL | logstashserver.coralogix.com                                                     | logstash.app.coralogix.in                                                        | logstashserver.coralogix.us                              |

## Validation

Test Filebeat by running it and monitoring the logs.

1. Modify the user credentials in `filebeat.yml` and specify a user who is authorized to publish events.

```bash
sudo chown root filebeat.yml
```

1. By default, Filebeat sends all of its output to Syslog. When you run Filebeat in the foreground, you can use the `-e` command line flag to redirect the output to standard error instead, as in the example below.

```bash
sudo ./filebeat -e
```

## Support

**Need help?**

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us **via our in-app chat** or by sending us an email at [support@coralogix.com](mailto:support@coralogix.com).