Learn more about Streama© – the foundational technology behind our stateful streaming data platform. Learn More

Coralogix Kubernetes Operator (cx-operator)

The Coralogix Kubernetes Operator or (cx-operator) makes it possible to configure Coralogix through custom resources. These custom resources behave just like any other Kubernetes resource; their main benefit is that they can be packaged and deployed alongside services deployed in the cluster (through a Helm chart, for example), and their definitions can be managed by source code management tools. This also means that the process for modifying Coralogix rules and alerts can be the standard code review process applied to application changes. This is a massive win on all fronts: better auditing, better change management, easier rollbacks, and more.

The cx-operator supports two types of custom resources:

  • Rule group sets (rulegroupsets.coralogix.com)
  • Alert sets (alertsets.coralogix.com)

Requirements

Kubernetes logging integrated with Coralogix. The tutorial can be found here.

Installation

The Coralogix Kubernetes Operator can be installed to the Kubernetes cluster with Helm.

The first step is to add cx-operator’s repository:

helm repo add cx-operator https://coralogix.github.io/cx-operator/

To see if that worked, search for the cx-operator chart:

helm search repo cx-operator

It should find the latest published version of the Coralogix Kubernetes Operator:

NAME                   	CHART VERSION	APP VERSION	DESCRIPTION
cx-operator/cx-operator	0.3.5+ddf73a7	0.3.5      	Coralogix Kubernetes Operator

The operator communicates with the Coralogix API with the user’s API key. This can be generated in the Account settings menu:

In Settings –> Account, Choose ‘Alerts API Access’ option and generate new Alerts & Rules API key:

** Note that only admin users have access to the API, So the option above will be visible only to admin users.

Once you have the key, it has to be stored in a Kubernetes Secret in your cluster:

kubectl create secret generic coralogix-operator-secrets --from-literal=RULES_API_TOKEN=00000000-0000-0000-0000-000000000000

Just replace the 00000000-0000-0000-0000-000000000000 with the API key from the settings page mentioned above. This will be used both for defining log processing rules and alerts.

With the secret in place all that remains is to install cx-operator into the cluster:

  • For accounts in Europe:
helm install --set config.coralogixApi.host=grpc-api.coralogix.com cx-operator cx-operator/cx-operator
  • For accounts in India:
helm install --set config.coralogixApi.host=grpc-api.app.coralogix.in cx-operator cx-operator/cx-operator

In case you also want to install a Prometheus ServiceMonitor for the operator, add the --set serviceMonitor.create=true option to the install command.

Log parsing rules

Log parsing rules can be defined using the rulegroupsets.coralogix.com custom resource. Please read the log parsing rules tutorial for more information about how these rules work.

** Note that currently supported rules by cx-operator are: Parse, Extract, Extract JSON, Replace, Block.

To define one or more log processing rules with the Coralogix Kubernetes Operator, you have to create one (or more) rule group set resources.

The following example defines two rule groups:

apiVersion: "coralogix.com/v1"
kind: RuleGroupSet
metadata:
  name: test-rulegroupset-1
spec:
  rule-groups-sequence:
    - name: 'Operator Test Rules 1'
      matcher:
        applications:
          - app1
          - app2
      and-sequence:
        - or-group:
          - name: 'Delete prefix'
            enabled: true
            source-field: Text
            replace:
              rule: '.*{'
              dest-field: Text
              new-value: '{'
        - or-group:
            - name: 'Extract bytes and status'
              description: 'Extracting bytes and status from message field'
              enabled: true
              source-field: Text
              extract:
                rule: '.*'
          - name: 'Worker to category'
            enabled: true
            source-field: worker
            json-extract:
              dest-field: Category

    - name: 'Operator Test Rules 2'
      matcher: {}
      enabled: false
      and-sequence:
        - or-group:
            - name: 'Block 28000'
              description: 'Block 28000 pg error'
              enabled: true
              source-field: field1
              block:
                rule: 'sql_error_code\s*=\s*28000'
                keep-blocked-logs: false

The exact, up-to-date definition of this custom resource can be found here.

The top level rule-groups-sequence is a list of rule groups to create. Each of these has the following properties:

Field nameRequired?Description
namerequiredThe name of the rule group
descriptionoptionalTextual description of the rule group
enabledoptionalEnabled state
hiddenoptionalControls the hidden state of the rule group
matcherrequiredA set of log matching rules, see below
and-sequencerequiredThe list of rules belonging to the rule group

The matcher block can have the following items:

Field nameRequired?Description
applicationsoptionalList of application names to apply the rule group to
subsystemsoptionalList of subsystem names to apply the rule group to
severitiesoptionalList of log severities (DEBUGVERBOSEINFOWARNINGERROR or CRITICAL to apply the rule group to

Each item of the and-sequence must be an or-group containing a list of rules. This hierarchy makes it possible to express complex rules in the form of “(Rule1 or Rule2) and Rule3 and (Rule4 or Rule5 or Rule6)”.

** Note the and-sequence is mandatory even if there is only one group of rules.

Rules have the following properties:

Field nameRequired?Description
namerequiredThe name of the rule
descriptionoptionalTextual description of the rule
source-fieldrequiredThe field of the log entry which is used by the rule
enabledoptionalThe enabled state of the single rule

Additionally to these there must be exactly one of the following parameters:

  • extract with a field rule defines an extract rule that copies information from the source field to new fields.
  • json-extract with a field dest-field (one of CATEGORYCLASSNAMEMETHODNAMETHREADID or SEVERITY) defines a JSON extract rule
  • replace defines a replace rule with dest-field selecting the target field, rule containing the regular expression and new-value the value to set the destination field to.
  • parse defines a parse rule with dest-field selecting the target field and rule containing the regular expression.
  • allow defines an expression in rule that selects log entries to not be blocked. The optional keep-blocked-logs field enables storing the blocked entries on S3.
  • block is the opposite of allow, it defines an expression in rule that selects log entries to be blocked. The optional keep-blocked-logs field enables storing the blocked entries on S3.

To install the rule group set, apply it to the cluster:

kubectl apply -f test-rulegroupset-1.yaml

Then verify that the custom resource has been stored:

kubectl get rulegroupsets

Alerts

Alerts can be defined using the alertsets.coralogix.com. One AlertSet resource can contain any number of alerts. To learn more about the different alert types, read the following tutorials:

The following example defines a single alert:

apiVersion: "coralogix.com/v1"
kind: AlertSet
metadata:
  name: test-alertset-1
spec:
  alerts:
    - name: test-alert-2
      description: "Testing the alerts operator"
      isActive: false
      severity: WARNING
      filters:
        filterType: TEXT
        severities:
          - ERROR
          - CRITICAL
        metadata:
          applications:
            - production
          subsystems:
            - my-app
            - my-service
        text: "authentication failed"
      condition:
        type: MORE_THAN
        parameters:
          threshold: 120
          timeframe: 10Min
          groupBy: host
      notifications:
        emails:
          - security@mycompany.com
          - mgmt@mycompany.com
        integrations: []

The exact, up-to-date definition of this custom resource can be found here.

Each alert in the alert set can have the following fields:

Field nameRequired?Description
namerequiredName of the alert
descriptionoptionalTextual description of the alert
isActiverequiredDetermines whether the alert is enabled or not
severityrequiredOne of INFOWARNING or CRITICAL
expirationoptionalThe expiration date for this alert
conditionoptionalSpecifies the condition of triggering the alert. Details below.
notificationsoptionalDefines email and webhook notifications. Details below.
filtersoptionalFiltering options, queries and ratio settings. Details below.
notifyEveryoptionalSupress time for the alert, in seconds
activeWhenoptionalSpecifies the time frames the alert is active. Details below.
notificationPayloadFiltersoptionalList of fields to be included in the alert message

Conditions

The optional condition block has one required field, type which can be:

  • IMMEDIATE
  • LESS_THAN
  • MORE_THAN
  • MORE_THAN_USUAL
  • NEW_VALUE

For all types except IMMEDIATE a second field is necessary, called properties, with the following contents:

Field nameRequired?Description
thresholdoptionalThe threshold parameter, defaults to 0
timeframerequiredThe valid timeframes depend on the condition type. For NEW_VALUE alerts it can be 12H..3M, otherwise 5Min..24H
groupByrequired for NEW_VALUESpecifies a field to aggregate into a histogram

Notifications

The optional notifications block specifies the recipients for the alert.

Field nameRequired?Description
emailsoptionalA list of e-mail addresses
integrationsoptionalA list of webhook integration names

See the Alert webhooks tutorial for more information on how to define these integrations.

Filters

The optional filters block contains several options to further customize the alert.

Field nameRequired?Description
filterTyperequiredOne of TEXTTEMPLATE or RATIO
severitiesoptionalA list of log severities to include. Each item must be one of DEBUGVERBOSEINFOWARNINGERROR or CRITICAL.
metadataoptionalA set of filters based on metadata. Details below.
aliasoptionalAn alias for the first query in case of ratio alerts. Default is “Query 1”
textoptionalThe filter query
ratioAlertsrequired if filterType is RATIOSpecifies the ratio alert. See below.

The metadata block can contain the following options, each one as a list of strings:

  • categories
  • applications
  • subsystems
  • computers
  • classes
  • methods
  • ipAddresses

For ratio alerts the ratioAlerts field contains further configuration possibilities. It is a list of ratio alert specifications, each having the following fields:

Field nameRequired?Description
aliasoptionalQuery alias, defaults to “Query N”
textoptionalThe filter query string
severitiesoptionalA list of log severities to include. Each item must be one of DEBUGVERBOSEINFOWARNINGERROR or CRITICAL.
applicationsoptionalA list of application names to include.
subsystemsoptionalA list of subsystem names to include

Active when

The optional activeWhen block has a single property, timeframes, which is a list of timeframe definitions. Each timeframe definition consists of:

Field nameRequired?Description
daysOfWeekoptionalA list of days the alert should be active. Each item must be one of MONDAYTUESDAYWEDNESDAYTHURSDAYFRIDAYSATURDAY or SUNDAY.
rangeoptionalA time range with two fields, start and end, both containing a time value in the format of HH:MM:SS.

Alert sets installed through Kubernetes can be queried using the alertsets name:

kubectl get alertsets