Coralogix Kubernetes Operator (cx-operator)

The Coralogix Kubernetes Operator or (cx-operator) makes it possible to configure Coralogix through custom resources. These custom resources behave just like any other Kubernetes resource; their main benefit is that they can be packaged and deployed alongside services deployed in the cluster (through a Helm chart, for example), and their definitions can be managed by source code management tools. This also means that the process for modifying Coralogix rules and alerts can be the standard code review process applied to application changes. This is a massive win on all fronts: better auditing, better change management, easier rollbacks, and more.

The cx-operator supports two types of custom resources:

• Rule group sets (rulegroupsets.coralogix.com)
• Alert sets (alertsets.coralogix.com)

Requirements

Kubernetes logging integrated with Coralogix. The tutorial can be found here.

Installation

The Coralogix Kubernetes Operator can be installed to the Kubernetes cluster with Helm.

The first step is to add cx-operator’s repository:

helm repo add cx-operator https://coralogix.github.io/cx-operator/

To see if that worked, search for the cx-operator chart:

helm search repo cx-operator

It should find the latest published version of the Coralogix Kubernetes Operator:

NAME                   	CHART VERSION	APP VERSION	DESCRIPTION
cx-operator/cx-operator	0.3.5+ddf73a7	0.3.5      	Coralogix Kubernetes Operator

The operator communicates with the Coralogix API with the user’s API key. This can be generated in Data Flow –> API Keys. Choose “Alerts, Rules and Tags API Key” option and generate new “Alerts, Rules and Tags API Key”:

** Note that only admin users have access to the API, So the option above will be visible only to admin users.

Once you have the key, it has to be stored in a Kubernetes Secret in your cluster:

kubectl create secret generic coralogix-operator-secrets --from-literal=RULES_API_TOKEN=00000000-0000-0000-0000-000000000000

Just replace the 00000000-0000-0000-0000-000000000000 with the API key from the API Keys page mentioned above. This will be used both for defining log processing rules and alerts.

With the secret in place all that remains is to install cx-operator into the cluster:

• For accounts in Europe:

helm install --set config.coralogixApi.host=grpc-api.coralogix.com cx-operator cx-operator/cx-operator

• For accounts in India:

helm install --set config.coralogixApi.host=grpc-api.app.coralogix.in cx-operator cx-operator/cx-operator

In case you also want to install a Prometheus ServiceMonitor for the operator, add the --set serviceMonitor.create=true option to the install command.

Log parsing rules

Log parsing rules can be defined using the rulegroupsets.coralogix.com custom resource. Please read the log parsing rules tutorial for more information about how these rules work.

** Note that currently supported rules by cx-operator are: Parse, Extract, Extract JSON, Replace, Block.

To define one or more log processing rules with the Coralogix Kubernetes Operator, you have to create one (or more) rule group set resources.

The following example defines two rule groups:

apiVersion: "coralogix.com/v1"
kind: RuleGroupSet
metadata:
  name: test-rulegroupset-1
spec:
  rule-groups-sequence:
    - name: 'Operator Test Rules 1'
      matcher:
        applications:
          - app1
          - app2
      and-sequence:
        - or-group:
          - name: 'Delete prefix'
            enabled: true
            source-field: Text
            replace:
              rule: '.*{'
              dest-field: Text
              new-value: '{'
        - or-group:
            - name: 'Extract bytes and status'
              description: 'Extracting bytes and status from message field'
              enabled: true
              source-field: Text
              extract:
                rule: '.*'
          - name: 'Worker to category'
            enabled: true
            source-field: worker
            json-extract:
              dest-field: Category

    - name: 'Operator Test Rules 2'
      matcher: {}
      enabled: false
      and-sequence:
        - or-group:
            - name: 'Block 28000'
              description: 'Block 28000 pg error'
              enabled: true
              source-field: field1
              block:
                rule: 'sql_error_code\s*=\s*28000'
                keep-blocked-logs: false

The exact, up-to-date definition of this custom resource can be found here.

The top-level rule-groups-sequence is a list of rule groups to create. Each of these has the following properties:

The matcher block can have the following items:

Each item of the and-sequence must be an or-group containing a list of rules. This hierarchy makes it possible to express complex rules in the form of “(Rule1 or Rule2) and Rule3 and (Rule4 or Rule5 or Rule6)”.

** Note the and-sequence is mandatory even if there is only one group of rules.

Rules have the following properties:

Additionally, to these there must be exactly one of the following parameters:

• extract with a field rule defines an extract rule that copies information from the source field to new fields.
• json-extract with a field dest-field (one of CATEGORY, CLASSNAME, METHODNAME, THREADID or SEVERITY) defines a JSON extract rule
• replace defines a replace rule with dest-field selecting the target field, rule containing the regular expression and new-value the value to set the destination field to.
• parse defines a parse rule with dest-field selecting the target field and rule containing the regular expression.
• allow defines an expression in rule that selects log entries to not be blocked. The optional keep-blocked-logs field enables storing the blocked entries on S3.
• block is the opposite of allow, it defines an expression in rule that selects log entries to be blocked. The optional keep-blocked-logs field enables storing the blocked entries on S3.

To install the rule group set, apply it to the cluster:

kubectl apply -f test-rulegroupset-1.yaml

Then verify that the custom resource has been stored:

kubectl get rulegroupsets

Alerts

Alerts can be defined using the alertsets.coralogix.com. One AlertSet resource can contain any number of alerts. To learn more about the different alert types, read the following tutorials:

• User defined alerts
• Ratio alerts
• Dynamic alerts

The following example defines a single alert:

apiVersion: "coralogix.com/v1"
kind: AlertSet
metadata:
  name: test-alertset-1
spec:
  alerts:
    - name: test-alert-2
      description: "Testing the alerts operator"
      isActive: false
      severity: WARNING
      filters:
        filterType: TEXT
        severities:
          - ERROR
          - CRITICAL
        metadata:
          applications:
            - production
          subsystems:
            - my-app
            - my-service
        text: "authentication failed"
      condition:
        type: MORE_THAN
        parameters:
          threshold: 120
          timeframe: 10Min
          groupBy: host
      notifications:
        emails:
          - [email protected]
          - [email protected]
        integrations: []

The exact, up-to-date definition of this custom resource can be found here.

Each alert in the alert set can have the following fields:

Conditions

The optional condition block has one required field, type which can be:

• IMMEDIATE
• LESS_THAN
• MORE_THAN
• MORE_THAN_USUAL
• NEW_VALUE

For all types except IMMEDIATE a second field is necessary, called properties, with the following contents:

Notifications

The optional notifications block specifies the recipients for the alert.

See the Alert webhooks tutorial for more information on how to define these integrations.

Filters

The optional filters block contains several options to further customize the alert.

The metadata block can contain the following options, each one as a list of strings:

• categories
• applications
• subsystems
• computers
• classes
• methods
• ipAddresses

For ratio alerts the ratioAlerts field contains further configuration possibilities. It is a list of ratio alert specifications, each having the following fields:

Active when

The optional activeWhen block has a single property, timeframes, which is a list of timeframe definitions. Each timeframe definition consists of:

Alert sets installed through Kubernetes can be queried using the alertsets name:

kubectl get alertsets