Rsyslog

  • coralogix
  • November , 2016,11

Determining syslog type

Coralogix supports both rsyslog and syslog-ng.

  1. Open your terminal window and type:
ls -d /etc/*syslog*

If you see rsyslog.d, you are using rsyslog. If you see syslog-ng, you are using syslog-ng.

If you don’t see any of these options then please install rsyslog or syslog-ng. Most linux distributions already have one of these syslog packages so you should refer to documentation of your linux distribution for installation guidelines.

rsyslog configuration

  • Open your terminal window and open the file rsyslog.conf with your favorite editor.
#!bash

vi /etc/rsyslog.conf
  • Locate a line containing the parameter $RepeatedMsgReduction. If it is configured to ‘on’ then please turn it off. If the line is commented then please uncomment it. If you can’t find this parameter then you should add it. After the modification it should look like this:
#!bash

$RepeatedMsgReduction off
  • Download the coralogix rsyslog configuration file rsyslog.conf (right click link + save as) and save it in /etc/rsyslog.d/ folder.
#!bash

cd /etc/rsyslog.d && wget https://syslogfiles.blob.core.windows.net/syslogfiles/coralogix.rsyslog.conf
#!bash

vi /etc/rsyslog.d/coralogix.rsyslog.conf

Template configuration

  • Navigate to the template section and update the CoralogixSyslogFormat parameter with your specific values. You should change only the values in BOLD:
#!bash
#*****************************************************************#
#                        TEMPLATE SECTION                         #
#*****************************************************************#

$template CoralogixSyslogFormat,"{\"fields\": {\"private_key\":\"YOUR COMPANY KEY\",\"company_id\":\"YOUR COMPANY ID\",\"app_name\":\"YOUR APPLICATION NAME\",\"subsystem_name\":\"YOUR APPLICATION SUBSYSTEM NAME\"},\"message\": {        \"message\":\"%msg%\",\"program_name\":\"%programname%\",\"pri_text\":\"%pri-text%\",\"hostname\":\"%HOSTNAME%\",\"tag\":\"%syslogtag%\"}}\n"

[YOUR COMPANY ID]: A unique ID which represents your company, this ID will be sent to your mail once you register to Coralogix.

[YOUR COMPANY KEY]: You can locate your company key in the Coralogix dashboard. Please navigate to Settings->SEND YOUR LOGS

[YOUR APPLICATION NAME]: The Application name parameter allows you to split between the different sources of your data, whether it’s different environments or complete different applications.

[YOUR APPLICATION SUBSYSTEM NAME]: Your application probably has multiple subsystems, for example: Backend servers, Middleware, Frontend servers etc. in order to help you examine the data you need, inserting the subsystem parameter is vital.

If you have several applications or subsystem components writing to the same syslog then you should create this template for each and one of them and give each a unique name. For instance, if you have an application with the name myapp and under that application you have 2 subsystems: mydal and and myclient (2 different processes running on the same host). An appropriate template configuration would be:

#!bash

$template CoralogixSyslogForma1,"{\"fields\": {\"private_key\":\"530e925d-be9e-****-****-75884f54efbe\",\"company_id\":\"****\",\"app_name\":\"prod\",\"subsystem_name\":\"nginx\"},\"message\": {        \"message\":\"%msg%\",\"program_name\":\"%programname%\",\"pri_text\":\"%pri-text%\",\"hostname\":\"%HOSTNAME%\",\"tag\":\"%syslogtag%\"}}\n"

Filter configuration

  • Navigate to the filter section of the file
#*****************************************************************#
#                        FILTER SECTION                           #
#*****************************************************************#

#Filter messages and send only the relevant one
#For more information and other filter options please refer to:
#http://www.rsyslog.com/doc/v8-stable/configuration/filters.html

#This will filter messages and send only the one with program name equal to: myApp
#:programname, isequal, "myapp" 

#This will filter messages and send only the one with facility equal to: user
#:syslogfacility-text, isequal, "user"

You should configure syslog to send logs only from your application rather then the entire messages coming from your linux OS. Using rsyslog filters you are basically forwarding only those messages that successfully passed your filter. You can define many different filters. For example:

To filter only application with the name myapp:

#!bash

:programname, isequal, "myapp"

To filter only applications writing to facility user:

#!bash

:syslogfacility-text, isequal, "user"

You can also filter by other parameters and you can use regular expressions as well. For more information on filtering please refer to: rsyslog filters

In case you do want to send the entire syslog data, you can ignore the filter section.

Destination configuration

  • Navigate to destination section.
#!bash

#*****************************************************************#
#                      DESTINATION SECTION                        #
#*****************************************************************#
#Send with UDP
*.* @syslogserver.coralogix.com:5140;CoralogixSyslogFormat
#Send with TCP
#*.* @@syslogserver.coralogix.com:5140;CoralogixSyslogFormat
#Print messages locally. Great for debugging #*.* /var/log/messages;CoralogixSyslogFormat

You can configure rsyslog to send logs via TCP or UDP protocol. By default the configuration file is configured for UDP. If you prefer TCP then please comment the UDP line and uncomment the TCP one. Coralogix is using port 514 for UDP and 1514 for TCP

In addition you can redirect your syslog messages to your local file. This is useful if you want to see the exact data that is being sent to Coralogix. This option is great for debugging. If you are having troubles sending your syslog data to Coralogix, then you should first check logs are written locally. Another good example can be in case you want to filter your syslog data based on your application name but you are not sure the exact name of the process. If you want to use this option then just uncomment this line:

#!bash

*.* /var/log/messages;CoralogixSyslogFormat

Here is an example for the log output of an application myapp sending log: Hello World!:

#!bash

[email protected]:~$ tail -f /var/log/messages

Nov 10 21:10:06 127.0.0.1/127.0.0.1 crx=1 crxversion=1 crxtype=syslog crxcompid=1
crxpkey=11111111-1111-1111-1111-1111111111 crxapp=myapp crxsubsys=mydal crxhostname=hostname1
crxtag='' <CRX.TIME_STAMP=1478812206820706> <CRX.PRI=daemon.err> <CRX.CATEGORY_REWRITE=''>
<CRX.SEVERITY_REWRITE=''> <CRX.MSG_REWRITE=''> <CRX.PROGRAM_NAME=my-app1> <CRX.MSG=Hello World!>

crxapp=myapp – This is the name of the application as you defined it in the template.

<CRX.PROGRAM_NAME=my-app1> – This is the process/program name that sent the log line to syslog.

If you defined several templates for each program/process name then instead of using one generic redirect rule:

#!bash

*.* @@syslogserver.coralogix.com:5140;CoralogixSyslogFormat

You should use a conditional redirect. For instance:

#!bash

if $programname == 'mydal' then @@syslogserver.coralogix.com:5140;CoralogixSyslogFormat1

if $programname == 'myclient' then @@syslogserver.coralogix.com:5140;CoralogixSyslogFormat2

By default syslog listens for messages on a LOCAL Unix domain socket. In case you are sending messages to your local syslog using TCP or UDP, you need to enable this option. Please read rsyslog source documentation to enable this option.

*** Save the file and restart rsyslog. The command to restart rsyslog daemon can vary from one Linux distribution to another but in most cases this would be:

#!bash

sudo service rsyslog restart

Testing configuration

  • At this point you should be ready to test your configuration. To send a test message you can use the linux logger command:
#!bash

logger -p info Hello World!

This should send a message “Hello World!” with severity info. If you enabled the option to redirect your syslog messages to a local file then you should see this message with the command:

#!bash

tail -f /var/log/messages

If you don’t see the message then please check your configuration.

Next, navigate to ‘Log Query’ menu in the Coralogix dashboard and press the Go button to search for the last 15 minutes logs. If you see your log then CONGRATULATIONS! You are now connected to Coralogix.

Still not viewing your logs ?, book your implementation session, and we’ll make sure your logs are right where they should be.

Docker syslog configuration

Docker provides several log drivers that can redirect console output logs to a log server. For a complete list of log drivers please refer to: Docker log drivers

This section describes how to work with Docker syslog driver and redirect your messages to Coralogix server.

  • Make sure that the host running your Docker container has a syslog daemon up and running.

ps aux | grep syslog

  • Configure your syslog to forward messages to Coralogix server. Please see section 2 for rsyslog or section 3 for syslog-ng.
  • Run your docker with the –log-driveroption, for instance:

docker run -d –log-driver=syslog ubuntu /bin/sh -c “while true; do echo hello world; sleep 1; done”

  • If you are using docker-compose V2 you can alternatively use:

my-container1:

  image: ubuntu

  entrypoint: /bin/sh -c “while true; do echo hello world; sleep 1; done”

  logging:

    driver: “syslog”

 

That’s it. Your logs should now appear in Coralogix dashboard.
Need help? We love to assist our customers, simply book your implementation session, and we will walk you through, step by step.

Signup to Coralogix

Related Posts:

WordPress Lightbox