Prometheus Alertmanager data ingestion

Open in ChatGPTOpen in Claude

Prometheus knows when a rule fired; Coralogix knows what the rest of the system was doing at the time. Forwarding Alertmanager notifications puts the two together, so alerting activity can be triaged and analyzed alongside your telemetry.

The Contextual Data integration generates the webhook URL; Alertmanager posts to it.

What you need

• Prometheus configured with Alertmanager, and alerting rules already defined.
• A Coralogix API key. You can create one during setup.
• An application and subsystem name to file the alerts under.

Generate the URL in Coralogix

1.

Open the Prometheus integration

Select Data Flow, then Contextual Data, then Prometheus, then Add, then Add new.

2.

Enter the integration details

Enter a Name. Supply Your API Key or select Create new key. Enter an App Name and Subsystem Name.

3.

Generate the URL

Select Generate URL and copy the result.

Shows the form that produces the webhook URL Alertmanager posts to.

Configure Alertmanager

4.

Check your rules

Confirm the rules in your alert file are valid before wiring anything up:

promtool check rules /path/to/rules.yml

5.

Add the Coralogix receiver

Open alertmanager.yml (usually at /etc/alertmanager/alertmanager.yml, though the path depends on your installation) and add the receiver, replacing Coralogix-url with the URL you generated:

route:
  receiver: Coralogix

receivers:
  - name: Coralogix
    webhook_configs:
      - url: Coralogix-url
        send_resolved: true

send_resolved: true forwards the resolution as well as the firing, so a Coralogix Case closes when the alert clears.

6.

Validate the configuration

Alertmanager refuses to start on an invalid configuration, so check it first:

amtool check-config alertmanager.yml

A success message lists what the file configures.

Example log

Each notification arrives with its status, labels, annotations, and the window it covers:

{
  "source_system": "prometheus",
  "prometheus": {
    "status": "resolved",
    "labels": {
      "alertname": "InstanceDown",
      "instance": "10.0.0.88:9100",
      "job": "node_exporter",
      "severity": "warning"
    },
    "annotations": {
      "description": "10.0.0.88:9100 of job node_exporter has been down for more than 5 minutes.",
      "summary": "Instance 10.0.0.88:9100 down"
    },
    "startsAt": "2021-03-19T14:59:56.248898138Z",
    "endsAt": "2021-03-19T16:28:11.248898138Z",
    "generatorURL": "http://CentosMachine:9090/graph?g0.expr=up+%3D%3D+0&g0.tab=1"
  }
}

Related resources

Prometheus Alert Manager triagePrometheus agentApplication and subsystem names