Coralogix provides seamless integration with Filebeat so you can send your logs from anywhere and parse them according to your needs.
Prerequisites
- Have Filebeat installed, for more information on how to install: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html
- Install our SSL certificate to your system for providing a secure connection. You can download it by the link: https://coralogix-public.s3-eu-west-1.amazonaws.com/certificate/ca.crt
General
Private Key – A unique ID which represents your company, this Id will be sent to your mail once you register to Coralogix.
Company Id – A unique number which represents your company. You can get your company id from the settings tab in the Coralogix dashboard.
Application Name – The name of your main application, for example, a company named “SuperData” would probably insert the “SuperData” string parameter or if they want to debug their test environment they might insert the “SuperData– Test”.
SubSystem Name – Your application probably has multiple subsystems, for example, Backend servers, Middleware, Frontend servers, etc. in order to help you examine the data you need, inserting the subsystem parameter is vital.
Installation
For a quick setup of Filebeat on your server, you can use prepared scripts.
Go to the folder with your Filebeat configuration file (filebeat.yml) and execute (as root):
deb
$ curl -sSL https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/scripts/install-deb.sh | bash
rpm
$ curl -sSL https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/scripts/install-rpm.sh | bash
This script will install Filebeat on your machine, prepare configuration and download Coralogix SSL certificates.
Note: If you want to install a specific version of Filebeat you should pass version number with environment variable before script run:
$ export FILEBEAT_VERSION=6.6.2
Configuration
On host machine
Open your Filebeat configuration file and configure it to use Logstash (Make sure you disable Elasticsearchoutput). For more information about configuring Filebeat to use Logstash please refer to https://www.elastic.co/guide/en/beats/filebeat/current/config-filebeat-logstash.html
Point your Filebeat to output to Coralogix Logstash server:
logstashserver.coralogix.com:5044
or if you want to use an encrypted connection (recommended):
logstashserver.coralogix.com:5015
In addition, you should add Coralogix configuration from the General section.
Here is a basic example of filebeat.yml:
#============================== Filebeat Inputs ===============================
filebeat.inputs:
- type: log
paths:
- "/var/log/your_app/your_app.log"
fields_under_root: true
fields:
PRIVATE_KEY: "YOUR_PRIVATE_KEY"
COMPANY_ID: YOUR_COMPANY_ID
APP_NAME: "APP_NAME"
SUB_SYSTEM: "SUB_NAME"
#----------------------------- Logstash output --------------------------------
output.logstash:
enabled: true
hosts: ["logstashserver.coralogix.com:5015"]
tls.certificate_authorities: ["<path to folder with certificates>/ca.crt"]
ssl.certificate_authorities: ["<path to folder with certificates>/ca.crt"]
Note: If you want to send all additional metadata, the fields_under_root option should be equals to true.
If you have multiline logs like:
2019-08-31 14:27:33 [main] ERROR Main - Exception
javax.management.RuntimeErrorException: null
at Main.main(Main.java:16) ~[bin/:na]
You can use multiline pattern:
filebeat.inputs:
- type: log
paths:
- "/var/log/your_app/your_app.log"
multiline:
pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \['
negate: true
match: after
The pattern should match the beginning of each record.
Docker
Build a Docker image with your filebeat.yml:
FROM docker.elastic.co/beats/filebeat:6.6.2
LABEL description="Filebeat logs watcher"
# Adding configuration file and SSL certificates for Filebeat
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
COPY ca.crt /etc/ssl/certs/Coralogix.crt
# Changing permission of configuration file
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml
# Return to deploy user
USER filebeat
Before deploying your container don’t forget to mount volume with your logs.
Kubernetes
Prerequisites
Before you will begin, make sure that you already have:
- Installed Kubernetes Cluster
- Enabled RBAC authorization mode support
Installation
First, you should create a Kubernetes secret with Coralogix credentials:
$ kubectl -n kube-system create secret generic filebeat-coralogix-account-secrets \
--from-literal=PRIVATE_KEY=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX \
--from-literal=COMPANY_ID=XXXX
You should receive something like:
secret "filebeat-coralogix-account-secrets" created
Then you need to create filebeat-coralogix-logger resources on your Kubernetes cluster with this manifests:
$ kubectl create -f https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/kubernetes/filebeat-coralogix-rbac.yaml
$ kubectl create -f https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/kubernetes/filebeat-coralogix-cm.yaml
$ kubectl create -f https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/kubernetes/filebeat-coralogix-secret.yaml
$ kubectl create -f https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/kubernetes/filebeat-coralogix-ds.yaml
$ kubectl create -f https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/kubernetes/filebeat-coralogix-svc.yaml
Output:
serviceaccount "filebeat-coralogix-service-account" created clusterrole "filebeat-coralogix-service-account-role" created clusterrolebinding "filebeat-coralogix-service-account" created configmap "filebeat-coralogix-config" created secret "filebeat-coralogix-certificate" created daemonset "filebeat-coralogix-daemonset" created service "filebeat-coralogix-service" created
Now filebeat-coralogix-logger collects logs from your Kubernetes cluster.
Here is the example of log record:
{
"cloud": {
"availability_zone": "projects/379343634745/zones/us-central1-a",
"instance_id": "7653580772456904060",
"instance_name": "gke-coralogix-test-default-pool-4d86c144-sbkd",
"machine_type": "projects/379343634745/machineTypes/n1-standard-1",
"project_id": "coralogix-test",
"provider": "gce"
},
"kubernetes": {
"container": {
"name": "prometheus-to-sd"
},
"labels": {
"k8s-app": "kube-dns",
"pod-template-hash": "989689126"
},
"namespace": "kube-system",
"node": {
"name": "gke-coralogix-test-default-pool-4d86c144-sbkd"
},
"pod": {
"name": "kube-dns-fdfbdf56b-jbbw2",
"uid": "56584469-534d-11e9-8bcd-42010a800179"
},
"replicaset": {
"name": "kube-dns-fdfbdf56b"
}
},
"@timestamp": "2019-03-31T00:45:53.973Z",
"@version": "1",
"host": {
"name": "filebeat-coralogix"
},
"beat": {
"hostname": "filebeat-coralogix-daemonset-98wxr",
"name": "filebeat-coralogix",
"version": "6.7.0"
},
"message": "E0331 00:45:53.970719 1 stackdriver.go:58] Error while sending request to Stackdriver Post /v3/projects/coralogix-test/timeSeries?alt=json: unsupported protocol scheme \"\"",
"tags": [
"kubernetes",
"containers",
"beats_input_codec_plain_applied"
]
}
Uninstall
If you want to remove filebeat-coralogix-logger from your cluster, execute this:
$ kubectl -n kube-system delete secret filebeat-coralogix-account-secrets
$ kubectl -n kube-system delete secret filebeat-coralogix-certificate
$ kubectl -n kube-system delete svc,ds,cm,clusterrolebinding,clusterrole,sa \
-l k8s-app=filebeat-coralogix-logger
