Fluentd

Coralogix provides seamless integration with Fluentd so you can send your logs from anywhere and parse them according to your needs.

Prerequisites

1. Sign up for a Coralogix account. Set up your account on the Coralogix domain corresponding to the region within which you would like your data stored.

2. Access your Coralogix private key.

3. Install Fluentd.

Parameters

Every configuration requires that you supply your Coralogix private key and the Coralogix Logs API endpoint associated with your Coralogix domain.

Configuration

The following section provides a number of configuration examples.

Generic Configuration

The example below collects all log files from the /var/log/ path, using the generic HTTP output plugin. We recommend this plugin due to its adjustability and exposed metrics. Other collection options can be found here.

<system>
	log_level info
</system>

<source>
	@type tail
	@id tail_var_logs
	@label @CORALOGIX
	path /var/log/*.log
	pos_file /var/logs/all.pos
        path_key path
	tag all
	read_from_head true
	<parse>
		@type none
	</parse>
</source>

<label @CORALOGIX>
	<filter **>
	@type record_transformer
	@log_level warn
	enable_ruby true
	auto_typecast true
	renew_record true
	<record>
		applicationName "Example_App"
		subsystemName "Example_Subsystem"
		text ${record.to_json}
	</record>
	</filter>

<match **>
	@type http
	@id http_to_coralogix
	endpoint "<Coralogix Logs API endpoint associated with your Coralogix domain>"
	headers {"private_key":"<your cluster's private key>"}
	retryable_response_codes 503
	error_response_as_unrecoverable false
	<buffer>
		@type memory
		chunk_limit_size 10MB
		compress gzip
		flush_interval 1s
		retry_max_times 5
		retry_type periodic
		retry_wait 2
	</buffer>
	<secondary>
		#If any messages fail to send they will be send to STDOUT for debug.
		@type stdout
	</secondary>
</match>
</label>

Configuration for Multiple Sources

The example below provides a Fluentd configuration that has several log file sources and a unique subsystem name for each source. It uses the “tag” option to specify a unique value in each source and a “tag” variable as subsystem. The result is several different tags depending on the source of the log.

<system>
  log_level info
</system>

<source>
  @type tail
  @id tail_path_1
  @label @CORALOGIX
  path /<path number 1>/*/*.log
  pos_file /<path number 1>/all.pos
        path_key path
  tag path_number_1
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<source>
  @type tail
  @id tail_path_2
  @label @CORALOGIX
  path /<path number 2>/*/*.log
  pos_file /<path number 2>/all.pos
        path_key path
  tag path_number_2
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<label @CORALOGIX>
  <filter **>
  @type record_transformer
  @log_level warn
  enable_ruby true
  auto_typecast true
  renew_record true
  <record>
    applicationName "<Application Name Here>"
    subsystemName ${tag}
    text ${record.to_json}
  </record>
  </filter>

<match **>
  @type http
  @id http_to_coralogix
  endpoint "<Coralogix Logs API endpoint associated with your Coralogix domain>"
  headers {"private_key":"xxxxxxxxxxxxxxxxxxx"}
  retryable_response_codes 503
  error_response_as_unrecoverable false
  <buffer>
    @type memory
    chunk_limit_size 10MB
    compress gzip
    flush_interval 1s
    retry_max_times 5
    retry_type periodic
    retry_wait 2
  </buffer>
  <secondary>
    #If any messages fail to send they will be send to STDOUT for debug.
    @type stdout
  </secondary>
</match>
</label>

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].