Fluentd

Coralogix provides seamless integration with Fluentd so you can send your logs from anywhere and parse them according to your needs.

Prerequisites

• Fluentd installed

Parameters

Every configuration requires that you supply your Coralogix Send-Your-Data API key (Api_Key) and the Coralogix REST API Singles endpoint associated with your Coralogix domain.

Configuration

The following section provides a number of configuration examples.

Generic Configuration

The example below collects all log files from the /var/log/ path, using the generic HTTP output plugin. We recommend this plugin due to its adjustability and exposed metrics. Other collection options can be found here.

<system>
	log_level info
</system>

<source>
	@type tail
	@id tail_var_logs
	@label @CORALOGIX
	path /var/log/*.log
	pos_file /var/logs/all.pos
        path_key path
	tag all
	read_from_head true
	<parse>
		@type none
	</parse>
</source>

<label @CORALOGIX>
	<filter **>
	@type record_transformer
	@log_level warn
	enable_ruby true
	auto_typecast true
	renew_record true
	<record>
		applicationName "Example_App"
		subsystemName "Example_Subsystem"
		text ${record.to_json}
	</record>
	</filter>

<match **>
	@type http
	@id http_to_coralogix
	endpoint "<Coralogix Logs API endpoint associated with your Coralogix domain>"
	headers {"authorization":"Bearer <Api_Key>"} 
	retryable_response_codes 503
	error_response_as_unrecoverable false
	<buffer>
		@type memory
		chunk_limit_size 10MB
		compress gzip
		flush_interval 1s
		retry_max_times 5
		retry_type periodic
		retry_wait 2
	</buffer>
	<secondary>
		#If any messages fail to send they will be send to STDOUT for debug.
		@type stdout
	</secondary>
</match>
</label>

Configuration for Multiple Sources

The example below provides a Fluentd configuration that has several log file sources and a unique subsystem name for each source. It uses the “tag” option to specify a unique value in each source and a “tag” variable as subsystem. The result is several different tags depending on the source of the log.

<system>
  log_level info
</system>

<source>
  @type tail
  @id tail_path_1
  @label @CORALOGIX
  path /<path number 1>/*/*.log
  pos_file /<path number 1>/all.pos
        path_key path
  tag path_number_1
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<source>
  @type tail
  @id tail_path_2
  @label @CORALOGIX
  path /<path number 2>/*/*.log
  pos_file /<path number 2>/all.pos
        path_key path
  tag path_number_2
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<label @CORALOGIX>
  <filter **>
  @type record_transformer
  @log_level warn
  enable_ruby true
  auto_typecast true
  renew_record true
  <record>
    applicationName "<Application Name Here>"
    subsystemName ${tag}
    text ${record.to_json}
  </record>
  </filter>

<match **>
  @type http
  @id http_to_coralogix
  endpoint "<Coralogix Rest API Singles endpoint associated with your Coralogix domain>"
  headers {"authorization":"Bearer <Api_Key>"}
  retryable_response_codes 503
  error_response_as_unrecoverable false
  <buffer>
    @type memory
    chunk_limit_size 10MB
    compress gzip
    flush_interval 1s
    retry_max_times 5
    retry_type periodic
    retry_wait 2
  </buffer>
  <secondary>
    #If any messages fail to send they will be send to STDOUT for debug.
    @type stdout
  </secondary>
</match>
</label>

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].