Our next-gen architecture is built to help you make sense of your ever-growing data Watch a 4-min demo video!

Back to All Docs

Auto-Generated Custom Enrichments Service

Last Updated: Jan. 05, 2023

Coralogix’s Security Traffic Analyzer (STA) collects data using internal enrichment services, for example, aws-context, k8s-context, and geographical information.

Then it sends over time those enrichments to Coralogix as CSV files per service so you’ll be able to enrich any of your logs with additional enriched fields – even if the source is not STA!

That way a connection and relations can be established between all types of logs sent to Coralogix’s platform which will improve readability, deeper understanding of contexts, and reduce investigation time dramatically over large content of logs from different sources.

All custom enrichment files can be found on Coralogix’s platform under Data Flow -> Data Enrichment section.

Enrichment Types

STA collects enrichments using several services which are running in the background.

To see the full list of STA’s enrichment modules, run the following command inside STA: sta-get-status-short

In the console, you’ll find all running services inside STA, and their current status. See the example below:

As you can see, all rows with the prefix: coralogix.sta.enrichment represents different services that handle STA’s enrichments.

Currently from the picture above we can see the following services:

  1. domains-tld-extract
  2. domain-stats
  3. dns-rbls
  4. unshorten-url
  5. domains-similarity
  6. freq-server
  7. geo
  8. nist-cpe
  9. aws-context
  10. k8s-context
  11. stats-info

Note: We are constantly improving and extending the STA, and more services might be added/modified.

Configuration

Structure

The configuration is located in sta.conf file that can be configured locally in STA or using Amazon S3 bucket.

{
    "sensors": {
        "custom_enrichment_producer": {
            "log_level": "TRACE" | "DEBUG" | "INFO" | "WARN" | "ERROR",
            "file_modification_minutes_threshold": int,
            "max_csv_rows": int,
            "enable": bool,
            "customized_services": [
                {
                    “name”: str,
                    “headers”: [str]
                }
            ],
            "disabled_for_services": [str],
            "other_services_enabled": bool,
        }
    }
}

Default Values

log_level: INFO

file_modification_minutes_threshold: 10

max_csv_rows: 9000

enable: true

customized_services: []

disabled_for_services: []

other_services_enabled: true

Variables explanation

NameTypeConstraintsDescription
log_levelPredefined stringPossible values:  "TRACE", "DEBUG""INFO", "WARN", "ERROR"The lowest log level to be shown in the console
file_modification_minutes_thresholdint1 ≤ x ≤ 60Number of minutes threshold of idleness before sending enrichments to Coralogix. This can happen when no new enrichments are found by the STA in a given period of time.
max_csv_rowsint2 ≤ x ≤ 10000Number of lines threshold per CSV file. if this threshold is reached, the file is sent to Coralogix, and a new file is created afterward.
enablebooleanenable/disable service. By default set to true.
customized_servicesArray of objectsPlease see 2 rows below for object representationLeave an empty array to include all services without specific configuration.
customized_services.namestrPredefined service names. Please see the section “Enrichment Types”.Names of the services to enable, if only the names are specified, only those services will be enabled. See “headers” below for additional configuration.
customized_services.headersArray of stringsBe sure that you know what headers you want as once you specify, only those will be searched, and others will be dropped.Representing the headers extracted into the service’s enrichment CSV – and only those. Leave an empty array to include all possible headers.
disabled_for_servicesArray of stringsPredefined service names. Please see the section “Enrichment Types”.Defines what services to exclude from CSV files.
other_services_enabledbooleanEnable other services that are not specified in the variable: customized_services. Find additional information and use cases below.

Use-Cases

For a better understanding of how those variables interact with STA, let’s see some use cases below:

Let’s assume that we want to disable aws-cotext enrichment service from sending CSVs to Coralogix. Our configuration should be as followed:

{
    "sensors": {
        "custom_enrichment_producer": {
            "disabled_for_services": [“aws-context],
        }
    }
}

As mentioned, the other values will receive their default values and only the specified service will be disabled.

Now let’s assume that we want to configure geo with specific headers: country, zone, coordinate.x, coordinate.y. In addition, we want to enable only aws-context and k8s-context services without configuration. Our configuration should be as followed:

{
    "sensors": {
        "custom_enrichment_producer": {
          "customized_services": [
            {
              “name”:”geo”,
              “headers”: [“country”,”zone”,”coordinate.x”, “coordinate.y”]
            },
            {
              “name”:”aws-context”,
              “headers”: []
            },
            {
              “name”:”k8s-context”,
              “headers”: []
            }
          ],
        "disabled_for_services": [],
        "other_services_enabled": false,
        }
    }
}

Finally, let’s assume again that we want to configure geo with specific headers: country, zone, coordinate.x, coordinate.y. In addition, we want all other services to enrich without configuration except the service nist-cpe which should be disabled. Our configuration should be as followed:

{
   "sensors": {
       "custom_enrichment_producer": {
          "customized_services": [
            {
              “name”:”geo”,
              “headers”: [“country”,”zone”,”coordinate.x”, “coordinate.y”]
            }
          ],
          "disabled_for_services": [“nist-cpe”],
          "other_services_enabled": true,
       }
   }
}

On this page