Skip to content

Rule Types

Coralogix offers several types of rules, each serving a different purpose in log parsing. Below you will find a full explanation of each rule type with an example use case.

Parsing Rules supports all of the following rule types:

For an overview of how parsing rules work, including rule groups, rule order, and AND/OR logic, see Log parsing rules.

BLOCK

BLOCK allows you to filter out incoming logs using a RegEx.

BLOCK matching logs

Original log:

{
  "transaction_ID": 12543,
  "worker": "A23",
  "message": "sql_error_code=28000, something went wrong"
}

RegEx (Block Rule):

sql_error_code=28000

Resulting log (if the log is blocked):

No log is passed through; the log is blocked and not processed further.

BLOCK non-matching logs

Original log:

{
  "transaction_ID": 12543,
  "worker": "A23",
  "message": "sql_error_code=20000, something went wrong"
}

RegEx (Block Rule):

sql_error_code=28000

Resulting log (if non-matching logs are blocked):

{
  "transaction_ID": 12543,
  "worker": "A23",
  "message": "sql_error_code=20000, something went wrong"
}

In this case, the log is blocked because it does not match the pattern (sql_error_code=28000), and it will be discarded from further processing.

Options:

  • Block all matching logs: Blocks any log that matches the Rule Matcher and BLOCK Rule (e.g., blocks logs containing sql_error_code=28000).
  • Block all non-matching logs: Blocks any log that does not match the Rule Matcher and BLOCK Rule (e.g., blocks all logs except those containing sql_error_code=28000).
  • View blocked logs in LiveTail: If this option is enabled, blocked logs are archived and visible in LiveTail as low-priority logs. Only 15% of low-priority log volume is counted against your team’s quota.

Note:

Even if you select the View blocked logs in LiveTail option, the log is dropped from the pipeline, and no downstream parsing rules will be applied to it.

Tips & tricks

BLOCK is particularly significant because it halts processing for any log that matches its conditions. For instance:

  • If a Block Rule is placed too early, it could stop essential transformations from being applied later in the pipeline.

Example:

  • Log Input:

    {"message": "test-message", "key2": "98a35"}

  • Block Rule:

    If this BLOCK rule is applied before any transformations, the log will exit the pipeline immediately without being processed further.

    .*98a35.*

To avoid this, ensure that BLOCK rules are carefully placed, allowing necessary transformations to occur beforehand. For optimal efficiency, BLOCK rules should still be prioritized early in the pipeline when no further transformations are required.

EXTRACT

EXTRACT retains the original log and adds fields at the root level based on the extraction.

Original log:

{"level":"INFO", "message": "200 bytes sent status is OK"}

RegEx:

message"\\\\s*:\\\\s*"(?P<bytes>\\\\d+)\\\\s*.*?status\\\\sis\\\\s(?P<status>[^"]+)

Resulting log with a structured original log:

{
  "level": "INFO",
  "message": "200 bytes sent status is OK",
  "bytes": "200",
  "status": "OK"
}

Resulting log with an unstructured original log:

{
  "bytes": "200",
  "text": "\\\\"level\\\\":\\\\"INFO\\\\", \\\\"message\\\\": \\\\"200 bytes sent status is OK\\\\"",
  "status": "OK"
}

JSON EXTRACT

JSON EXTRACT extracts a key's value from a JSON log and uses it to overwrite a field.

  • Source Field: Always text.
  • JSON Key Field: The field name or path from which to extract the value.
  • Destination Field: The field to overwrite.

Simple field extraction

Original Log:

{
  "transaction_ID": 12543,
  "worker": "A23",
  "message": "success"
}

JSON Key Field: worker

Destination Field: category

Resulting Log:

{
  "transaction_ID": 12543,
  "worker": "A23",
  "message": "success",
  "category": "A23"
}

In this example, the value of the worker key ("A23") is extracted and used to overwrite the category field.

Nested field extraction

Original Log:

{
  "transaction_ID": 54321,
  "user": "john_doe",
  "status": "OK",
  "metadata": {
    "region": "US-East",
    "app_version": "1.2.3"
  }
}

JSON Key Field: metadata.region

Destination Field: text.region

Resulting Log:

{
  "transaction_ID": 54321,
  "user": "john_doe",
  "status": "OK",
  "metadata": {
    "region": "US-East",
    "app_version": "1.2.3"
  },
  "region": "US-East"
}

Here, the region value from the nested metadata object is extracted and added as a new top-level field (region).

PARSE

PARSE uses RegEx-named capture groups. These groups become the parsed log fields, and the value associated with each group becomes the field’s value. The RegEx doesn’t have to match the entire log; only the named capture groups (the values within < >) and their captured values are included in the reconstructed log.

Original log:

sock=client at=warning code=H27 desc="Client Request Interrupted" method=POST path="/submit/" host=myapp.herokuapp.com fwd=17.17.17.17 dyno=web.1 connect=1ms service=0ms status=499 bytes=0

RegEx:

^(sock=)?(?P<sock>(\S*))\s*at=(?P<severity>\S*)\s*code=(?P<error_code>\S*)\s*desc="(?P<desc>[^"]*)"\s*method=(?P<method>\S*)\s*path="(?P<path>[^"]*)" host=(?P<host>\S*)\s* (request_id=)?(?P<request_id>\S*)\s*fwd="?(?P<fwd>[^"\s]*)"?\s*dyno=(?P<dyno>\S*)\s*connect=(?P<connect>\d*)(ms)?\s*service=(?P<service>\d*)(ms)?\s*status=(?P<status>\d*)\s* bytes=(?P<bytes>\S*)\s*(protocol=)?(?P<protocol>[^"\s]*)$

Resulting log:

{
  "sock": "client",
  "severity": "warning",
  "error_code": "H27",
  "desc": "Client Request Interrupted",
  "method": "POST",
  "path": "/submit/",
  "host": "myapp.herokuapp.com",
  "request_id": "",
  "fwd": "17.17.17.17",
  "dyno": "web.1",
  "connect": "1",
  "service": "0",
  "status": "499",
  "bytes": "0",
  "protocol": ""
}

REPLACE

REPLACE is commonly used to fix misformatted JSON logs or make structural modifications.

Fixing JSON format

Original log:

2020-08-07 {"status":"OK", "user":"John Smith", "ops":"J1"}

RegEx:

.*{

Replacement string:

{

Resulting log:

{"status":"OK", "user":"John Smith", "ops":"J1"}

Adding nested fields

Original log:

{"ops":"G1","user":"John Smith-2125 Sierra Ventura Dr.-Sunnyvale-CA-94054","status":"305"}

RegEx:

(.*user"):"([^-]*)-([^-]*)-([^-]*)-([^-]*)-([^-]*)",([^$]*)

Replacement string:

$1:{"name":"$2","address":"$3","city":"$4","state":"$5","zip":"$6"},$7

Resulting log:

{
  "ops": "G1",
  "user": {
    "name": "John Smith",
    "address": "2125 Sierra Ventura Dr.",
    "city": "Sunnyvale",
    "state": "CA",
    "zip": "94054"
  },
  "status": "305"
}

Warning:

The REPLACE rule matches a RegEx in your log and replaces it with a specified string. Be cautious, as this replacement can break the log's JSON validity. Always validate the output before applying.

REMOVE FIELDS

REMOVE FIELDS allows you to drop specific fields from JSON logs on the Coralogix side. This can help declutter logs by removing irrelevant or sensitive information.

Original Log:

{
  "transaction_ID": 12543,
  "worker": "A23",
  "message": "success",
  "irrelevant_field": "to_be_removed",
  "another_irrelevant_field": "remove_this_too"
}

Resulting Log After Applying the Rule:

{
  "transaction_ID": 12543,
  "worker": "A23",
  "message": "success"
}

Here, both irrelevant_field and another_irrelevant_field are removed from the log based on the specified RegEx.

STRINGIFY JSON FIELDS

STRINGIFY JSON FIELDS converts an array or object from JSON format to text while retaining searchability.

Original log:

{
  "sessionIssuer": {
    "appID": "A1",
    "env": "prod"
  }
}
  • Source Field: sessionIssuer
  • Destination Field: application

Resulting log with the source field removed:

{
  "application": "{\\\\"appID\\\\":\\\\"A1\\\\",\\\\"env\\\\":\\\\"prod\\\\"}"
}

Resulting log with the source field retained:

{
  "sessionIssuer": {
    "appID": "A1",
    "env": "prod"
  },
  "application": "{\\\\"appID\\\\":\\\\"A1\\\\",\\\\"env\\\\":\\\\"prod\\\\"}"
}

TIMESTAMP EXTRACT

TIMESTAMP EXTRACT replaces the Coralogix-assigned timestamp with a timestamp extracted from a field in the log, based on a defined format.

Original log:

{
  "transaction_ID": 12543,
  "worker": "A23",
  "message": "success",
  "time": "2021-01-11T15:04:05.000000+0100"
}
  • Source Field: time
  • Timestamp Format (strftime): %Y-%m-%dT%H:%M:%S.%f%z

Resulting Log:

The time field value replaces the default Coralogix timestamp for the log, while the log itself remains unchanged.

Using RegEx to extract the timestamp

If the timestamp is embedded within a larger string and needs to be extracted first, you can use a RegEx in a parsing rule. For example:

Original Log:

[2021-01-11T15:04:05.000000+0100] INFO: Task completed successfully.

RegEx for Parsing the Timestamp:

\[(?P<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{6}\+\d{4})\]

This RegEx captures the timestamp in the time field.

Resulting Log After Parsing:

{
  "time": "2021-01-11T15:04:05.000000+0100",
  "level": "INFO",
  "message": "Task completed successfully."
}

Once the timestamp is extracted, you can apply the TIMESTAMP EXTRACT rule to use the time field as the log's new timestamp.

PARSE JSON FIELD

PARSE JSON FIELD unescapes stringified logs and parses the inner JSON strings into fields.

Example

Original log:

{
  "server": "opa",
  "IBC": "45ML",
  "thread": "1201",
  "message": "{\"first_name\":\"John\", \"last_name\":\"Smith\", \"userID\":\"AB12345\", \"duration\":45}"
}
  • Source Field: text.message
  • Destination Field: text.parsed_data

RegEx:

"message"\\\\s*:\\\\s*"{\\\\s*\\\\\\\\"

Resulting log:

{
    "server": "opa",
    "IBC": "45ML",
    "thread": "1201",
    "message": "{\"first_name\":\"John\", \"last_name\":\"Smith\", \"userID\":\"AB12345\", \"duration\":45}",
    "parsed_data": {
      "first_name": "John",
      "last_name": "Smith",
      "userID": "AB12345",
      "duration": 45
    }
  }
Was this helpful?
Previous Log Parsing Rules
Next Rules Cheat Sheet
Home
User Guides
DataPrime
Integrations
OpenTelemetry
Developer Portal