Learn more about Streama© – the foundational technology behind our stateful streaming data platform. Learn More

Connect Logstash to AWS Kinesis

Coralogix provides integration to connect Logstash to AWS Kinesis , so you can send your logs from anywhere into Coralogix.

Prerequisites

Have Logstash installed, for more information on how to install: Installing Logstash.

Usage

You must provide the following four variables when creating a Coralogix logger instance.

Private Key – A unique ID that represents your company. The private key can be found under ‘settings’->’ send your logs’. It is located in the upper left corner.

Application Name – The name of your environment, for example, a company named “SuperData” would probably insert the “SuperData” string parameter or if they want to debug their test environment they might insert the “SuperData– Test”.

SubSystem Name – Your application probably has multiple components, for example: Backend servers, Middleware, Frontend servers etc. in order to help you examine the data you need, inserting the subsystem parameter is vital.

Region – The AWS region for Kinesis.

Installation

$ logstash-plugin install logstash-input-kinesis
$ logstash-plugin install logstash-output-coralogix

If you are not sure where the logstash-plugin is located, you can check this here.

Configuration

Open your Logstash configuration file and add AWS Kinesis input and Coralogix.

input {
  kinesis {
    kinesis_stream_name => "XXXXXXXX"
    region => "XX-XXXX-X"
    codec => json
  }
}

output {
    coralogix {
        config_params => {
            "PRIVATE_KEY" => "YOUR_PRIVATE_KEY"
            "APP_NAME" => "APP_NAME"
            "SUB_SYSTEM" => "SUB_NAME"
        }
        log_key_name => "message"
        timestamp_key_name => "@timestamp"
        is_json => true
    }
}

Input

  • kinesis_stream_name is mandatory.
  • region is optional (Default value is "us-east-1").

More information about how to setup Logstash Input Kinesis plugin: logstash-input-kinesis.

Output

Watch more information about our logstash-output-coralogix plugin.