Our next-gen architecture is built to help you make sense of your ever-growing data. Watch a 4-min demo video!

Back to All Docs

Parsing Rules API Parsing Rules API

Last Updated: Sep. 20, 2023

This guide will help you use our Parsing Rules API to create, read, update or delete parsing rules and rule groups via API.

In order to use the Parsing Rules API, first locate your:

  • Alerts, Rules, and Tags API key You can find your key in the Coralogix UI under Data Flow –> API Keys.
  • Coralogix domain

To send an external request, the request headers should contain the following Headers:

  • Content-Type: application/JSON
  • Authorization: Bearer {{Alerts, Rules, and Tags API key}}

Alerts, Rules, and Tags API key can be found under Data Flow –> API Keys.

NOTE – Only users with Admin privileges have access to the Rules API and keys.

This guide will help you use our Rules API to create, read, update or delete parsing rules and rule groups via API. Parsing rules are organized inside Rule Groups. Each group has a name and a set of parsing rules with a logical AND/OR relationship between them.

Logs are processed according to the order of the Rule Groups (top to bottom) and then by the order of rules within the Rule Group and according to the logical operators between them (AND/OR).

Group NameSub-Group / ConditionRule Name
My group 1My rule 1-1-a
OR
My rule 1-1-b
AND
My rule 1-2-a
OR
My rule 1-2-b
My group 2My rule 2-1-a

Create a Rule

To create a parsing rule, there is a need to define a rule group and the parsing rule itself:

Method: POST
URL: https://api.<domain>/api/v1/external/rule/rule-set
Body (the following fields are required/optional):

  • "name": (string)(mandatory) – the name of the rules group,
  • "description": (string)(optional – default is an empty string) – the description for the rules group,
  • "enabled": (boolean)(optional) – indicates rather the group is active or not when created – if fields is missing or value is empty – default is True,
  • "creator": (string)(optional – default is “Coralogix external API”) – the name of the group creator,
  • "order": (number)(optional) – the rule group order number. If it is not provided, the rule group will be added at the end of the list of rule groups. If the group with the provided order number already exists, new group will be added one below and the order will be changed to +1 automatically,
  • "ruleMatchers":(array of objects)(mandatory) – the conditions which the group of rules works on – application/subsystem/severity of logs. Each object in the array should contain:
    • "field": (string) can be any of the following – applicationName/ subsystemName/ severity,
    • "constraint": (string)- the value for the chosen field (if the field is severity, the value should be one: debug, verbose, info, warning, error, critical)
  • "rulesGroups":(array of objects)(optional) – It contains two fields:
    • "order": (number) – it is the number of the sub-group. If rules are under two different sub-groups then it means there is AND condition between them
    • "rules": (array of object) – each object is a rule

The rules object keys:

  • "name": (string)(mandatory) – the name of rule,
  • "description": (string)(optional – default is an empty string) – the description for the rule,
  • "enabled": (boolean)(optional) – indicates rather the rule is active or not when created – if fields is missing or value is empty – default is True,
  • "type": (string)(mandatory) – type of the rule (block, extract, parse, jsonextract, replace, timestampextract, removefields),
  • "sourceField": (string)(optional) – If not stated in the body then the default source field will be the log text. If you want to run the rule against any internal log JSON field use Text.field_name.field_name2…
  • "destinationField":(string)(optional) – If not stated in the body then the default destination field will be the log text. If you want to run the rule against any internal log JSON field use Text.field_name.field_name2…
  • "order": (number)(optional) – if empty then the rule is added at the end
  • "rule": (string)(mandatory – except in “removefields” and “timestampextract” – for more info look at restrictions) – the regex of the rule (must be a regex that can be compiled)
Rule TypeDescription / Restrictions
BLOCK, EXTRACTAPI request body cannot have: “ReplaceNewVal” OR “DestinationField”
PARSEAPI request body cannot have: “ReplaceNewVal”
JSONEXTRACTAPI request body cannot have: “ReplaceNewVal”. API request body must have: “DestinationField”:(string) – need to be one of: “category”, “className”, “methodName”, “severity”, “threadId”
REPLACEAPI request body must have: “ReplaceNewVal”
REMOVEFIELDSAPI request body must have: “rule”:(string) – comma-separated list of json fields you would like to remove
TIMESTAMPEXTRACTAPI request body cannot have: “rule” or “destinationField” fields. API request body must have:”timeFormat”(string) – you can see our suggestion for each type of standard, normally you will need to change the format to match your time field string pattern with the while comply to the standard “formatStandard”:(string) = need to be one of: “javasdf”, “golang”, “strftime”, “secondsts”, “millits”, “microts”, “nanots”). API request body should contain a meaningful value for “sourceField”: (string) – a value is not specified the default source files will be the text field and the rule will not work.You should set the value as Text.<json_key_with_time_field>)
STRINGIFY JSON FIELDConvert JSON object to JSON string. Find out more here.
PARSE JSON FIELDConvert JSON string to JSON object. Find out more here.

Examples

  1. A group with one parsing rule
{
    "name": "My test group with one rule - parse",
    "description": "It is an example how to create a group with a parsing rule",
    "enabled": true,
    "order": 1,
    "creator": "[email protected]",
    "ruleMatchers": [],
    "rulesGroups": [
        {
            "order": 1,
            "rules": [
                {
                    "name": "Test parse rule",
                    "description": "This is the first parsing rule",
                    "enabled": true,
                    "type": "parse",
                    "sourceField": "text",
                    "destinationField": "text",
                    "order": 1,
                    "rule": "^(?P<timestamp>\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\s*\\\\d{2}:\\\\d{2}:\\\\d{2}\\\\.\\\\d+)\\\\s+(?P<message>.*)$"
                }
            ]
        }
    ]
}
  1. A group with two parsing rules and an AND condition between them
{
    "name": "My test group with one rule per each sub-group",
    "description": "It is an example how to create a group with two rules with AND condition between them ",
    "enabled": true,
    "order": 1,
    "creator": "[email protected]",
    "ruleMatchers": [],
    "rulesGroups": [
        {
            "order": 1,
            "rules": [
                {
                    "name": "Test parse rule",
                    "description": "This is the first parsing rule",
                    "enabled": true,
                    "type": "parse",
                    "sourceField": "text",
                    "destinationField": "text",
                    "order": 1,
                    "rule": "^(?P<timestamp>\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\s*\\\\d{2}:\\\\d{2}:\\\\d{2}\\\\.\\\\d+)\\\\s+(?P<message>.*)$"
                }
            ]
        },
        {
            "order": 2,
            "rules": [
                {
                    "name": "Test replace rule",
                    "description": "This is the first replace rule",
                    "enabled": true,
                    "type": "replace",
                    "sourceField": "text.message",
                    "destinationField": "text.message",
                    "order": 1,
                    "rule": "host",
                    "replaceNewVal": "hostname"
                }
            ]
        }
    ]
}
  1. A group with 3 parsing rules
{
    "name": "My test group with three rules",
    "description": "It is an example how to create a group with 3 rules",
    "enabled": true,
    "order": 5,
    "creator": "[email protected]",
    "ruleMatchers": [],
    "rulesGroups": [
        {
            "order": 1,
            "rules": [
                {
                    "name": "Test parse rule",
                    "description": "This is the first parsing rule",
                    "enabled": true,
                    "type": "parse",
                    "sourceField": "text",
                    "destinationField": "text",
                    "order": 1,
                    "rule": "^(?P<timestamp>\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\s*\\\\d{2}:\\\\d{2}:\\\\d{2}\\\\.\\\\d+)\\\\s+(?P<message>.*)$"
                }
            ]
        },
        {
            "order": 2,
            "rules": [
                {
                    "name": "Test replace rule 1",
                    "description": "This is the first replace rule",
                    "enabled": true,
                    "type": "replace",
                    "sourceField": "text.message",
                    "destinationField": "text.message",
                    "order": 1,
                    "rule": "host",
                    "replaceNewVal": "hostname"
                },
                {
                    "name": "Test replace rule 2",
                    "description": "This is the second replace rule",
                    "enabled": true,
                    "type": "replace",
                    "sourceField": "text.message",
                    "destinationField": "text.message",
                    "order": 2,
                    "rule": "server",
                    "replaceNewVal": "hostname"
                }
            ]
        }
    ]
}

Get All Rules

To get all parsing rules:

Method: GET
URL: https://api.<domain>/api/v1/external/rules

In response, you will get the "companyRulesData" array with groups and rules.

Example

{
    "companyRulesData": [
        {
            "id": "20538afe-12d4-4032-a2d9-42b14e0036a1",
            "name": "My test group with one rule - parse",
            "description": "It is an example how to create a group with a parsing rule",
            "order": 1,
            "enabled": true,
            "creator": "[email protected]",
            "updatedAt": "2022-11-03 08:32:53",
            "createdAt": "2022-11-03 08:32:53",
            "ruleMatchers": null,
            "rulesGroups": [
                {
                    "id": "562cbdf3-5021-42ab-a85a-23c0e80db62b",
                    "order": 1,
                    "rules": [
                        {
                            "id": "f3ddb109-c80d-4a46-8aa9-98f2984fb2f7",
                            "name": "Test parse rule",
                            "description": "This is the first parsing rule",
                            "enabled": true,
                            "ruleMatchers": null,
                            "rule": "^(?P<timestamp>\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\s*\\\\d{2}:\\\\d{2}:\\\\d{2}\\\\.\\\\d+)\\\\s+(?P<message>.*)$",
                            "sourceField": "text",
                            "destinationField": "text",
                            "replaceNewVal": "",
                            "type": "parse",
                            "order": 1,
                            "keepBlockedLogs": false,
                            "deleteSource": false,
                            "escapedValue": false,
                            "overrideDest": false
                        }
                    ]
                }
            ],
            "hidden": false
        },
        {
            "id": "94b0a95b-994e-42ef-9c22-48d58b2e339e",
            "name": "My test group with one rule per each sub-group",
            "description": "It is an example how to create a group with two rules with AND condition between them ",
            "order": 2,
            "enabled": true,
            "creator": "[email protected]",
            "updatedAt": "2022-11-03 08:52:51",
            "createdAt": "2022-11-03 08:52:39",
            "ruleMatchers": null,
            "rulesGroups": [
                {
                    "id": "4dfde3e7-2791-4b34-aa0a-3edafe719ddb",
                    "order": 1,
                    "rules": [
                        {
                            "id": "5802ec65-dae5-4e20-bd1d-37dbf15e2630",
                            "name": "Test parse rule",
                            "description": "This is the first parsing rule",
                            "enabled": true,
                            "ruleMatchers": null,
                            "rule": "^(?P<timestamp>\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\s*\\\\d{2}:\\\\d{2}:\\\\d{2}\\\\.\\\\d+)\\\\s+(?P<message>.*)$",
                            "sourceField": "text",
                            "destinationField": "text",
                            "replaceNewVal": "",
                            "type": "parse",
                            "order": 1,
                            "keepBlockedLogs": false,
                            "deleteSource": false,
                            "escapedValue": false,
                            "overrideDest": false
                        }
                    ]
                },
                {
                    "id": "ca69375b-a869-4c6f-96eb-14a4e6c8a935",
                    "order": 2,
                    "rules": [
                        {
                            "id": "62e46153-a56a-4322-a1be-c4db871afaec",
                            "name": "Test replace rule",
                            "description": "This is the first replace rule",
                            "enabled": true,
                            "ruleMatchers": null,
                            "rule": "host",
                            "sourceField": "text.message",
                            "destinationField": "text.message",
                            "replaceNewVal": "hostname",
                            "type": "replace",
                            "order": 1,
                            "keepBlockedLogs": false,
                            "deleteSource": false,
                            "escapedValue": false,
                            "overrideDest": false
                        }
                    ]
                }
            ],
            "hidden": false
        }
    ]
}

As you can see, each group contains the id field which is the GROUPID.

The GROUPID can be also obtained from the Parsing Rules page.

  1. Go to Data Flow → Parsing Rules.
  2. Click on the rule group you would like to edit.
  3. Look at the URL in the browser address bar.
rule group id

Get a Rule Group

Query for parsing rules from the specific group.

Method: GET
URL: https://api.<domain>/api/v1/external/rule/rule-set/<GROUPID>

Example

https://api.coralogix.com/api/v1/external/rule/rule-set/20538afe-12d4-4032-a2d9-42b14e0036a1

{
    "id": "20538afe-12d4-4032-a2d9-42b14e0036a1",
    "name": "My test group with one rule - parse",
    "description": "It is an example how to create a group with a parsing rule",
    "order": 1,
    "enabled": true,
    "creator": "[email protected]",
    "updatedAt": "2022-11-03 08:32:53",
    "createdAt": "2022-11-03 08:32:53",
    "ruleMatchers": null,
    "rulesGroups": [
        {
            "id": "562cbdf3-5021-42ab-a85a-23c0e80db62b",
            "name": "",
            "order": 1,
            "enabled": true,
            "rules": [
                {
                    "id": "f3ddb109-c80d-4a46-8aa9-98f2984fb2f7",
                    "name": "Test parse rule",
                    "description": "This is the first parsing rule",
                    "enabled": true,
                    "ruleMatchers": null,
                    "rule": "^(?P<timestamp>\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\s*\\\\d{2}:\\\\d{2}:\\\\d{2}\\\\.\\\\d+)\\\\s+(?P<message>.*)$",
                    "sourceField": "text",
                    "destinationField": "text",
                    "replaceNewVal": "",
                    "type": "parse",
                    "order": 1,
                    "keepBlockedLogs": false,
                    "deleteSource": false,
                    "escapedValue": false,
                    "overrideDest": false
                }
            ],
            "type": "user"
        }
    ]
}

Modify Rules

Modify or update a group with one parsing rule.

Method: PUT
URL: https://api.<domain>/api/v1/external/rule/rule-set

To update any rule just perform GET with a GROUPID, modify all rules in the group from the response, and send it with PUT request.

Note: The parameter ruleMatchers at the group section is returned by the GET request. It needs to be changed to the empty array [] before sending the request.

Delete Rules

Delete a Group

Method: DELETE
URL: https://api.<domain>/api/v1/external/rule/rule-set/<GROUPID>

Delete a Parsing Rule

Method: DELETE
URL: https://api.<domain>/api/v1/external/rule/RULEID/group/GROUPID

where RULEID is the id of the rule.

Note: It is not possible to delete a sub-group of rules.

Export Multiple Rules and Groups

If there is a need to copy all rules from one team to another follow these steps:

  1. Export all groups and rules as described above in the section “Get all rules”.
  2. Create and send a new request with: a) the Rules API key of the target team, b) URL: https://api.<domain>/api/v1/external/rules/export c) The body should contain the JSON object you got from executing the GET request in step 1. Copy and paste the output of the GET request to the body of the POST request without changing anything. A message stating “Group and Rules transformed successfully” will be prompted once the transfer is complete.

NOTE: When moving rules from one account to another keep in mind that the rule groups you imported are concatenated after the existing rules groups, we do not override existing groups by default. If you want to override the existing groups you should first remove them entirely before importing new rule groups.

Legacy REST API Methods

The REST API methods below are the old ways to manage parsing rules. They do not support the order of rules and fields names may be different. They are still supported.

Groups API

Create a Group

Create a group which will keep your rules.

Method: POST
URL: https://api.<domain>/api/v1/external/group
Body (the following fields are required/optional):

  • "name": (string)(mandatory) – the name of the rules group,
  • "description": (string)(optional – default is an empty string) – the description for the rules group,
  • "enabled": (boolean)(optional) – indicates rather the group is active or not when created – if fields is missing or value is empty – default is True,
  • "creator": (string)(optional – default is “Coralogix external API”) – the name of the group creator,
  • "ruleMatchers":(array of objects)(mandatory) – the conditions which the group of rules work on – application/subsystem/severity of logs. Each object in the array should contain:
    • "field": (string) can be any of the following – applicationName/ subsystemName/ severity,
    • "constraint": (string)- the value for the chosen field (if the field is severity, the value should be one: debug, verbose, info, warning, error, critical)

Example request:

{
	"name": "My group 1",
	"description": "This is a test group",
	"enabled": true,
	"creator": "[email protected]",
	"ruleMatchers": [{
		"field": "applicationName",
		"constraint": "test_app"
	}, {
		"field": "subsystemName",
		"constraint": "test_sub"
	}, {
		"field": "severity",
		"constraint": "warning"
	}, {
		"field": "severity",
		"constraint": "error"
	}, {
		"field": "severity",
		"constraint": "critical"
	}]
}

Example response:

The rule was created at the bottom of all rule groups.

{
    "id": "73cac31e-225b-41d8-bc32-5a825d2b29a1",
    "name": "My group 1",
    "description": "This is a test group",
    "order": 20,
    "enabled": true,
    "creator": "[email protected]",
    "createdAt": "2022-11-02 14:47:56",
    "ruleMatchers": [
        {
            "field": "applicationName",
            "constraint": "test_app"
        },
        {
            "field": "subsystemName",
            "constraint": "test_sub"
        },
        {
            "field": "severity",
            "constraint": "warning"
        },
        {
            "field": "severity",
            "constraint": "error"
        },
        {
            "field": "severity",
            "constraint": "critical"
        }
    ],
    "rulesGroups": []
}

Read a Group

The first step is to create a group which will keep our rules.

Method: GET
URL: https://api.<domain>/api/v1/external/group/GROUPID
Body: Empty

Update a Group

The first step is to create a group that will keep our rules.

Method: PUT
URL: https://api.<domain>/api/v1/external/group/GROUPID
Body (the following fields are required/optional):

  • "name": (string)(mandatory) – the name of the rules group,
  • "Description": (string)(optional – default is an empty string) – the description for the rules group,
  • "enabled": (boolean)(optional) – indicates rather the group will be set to active or not when updating,
  • "ruleMatchers":(array of objects)(mandatory) – the conditions which the group of rules work on – application/subsystem/severity of logs. Each object in the array should contain:
    • "field": (string) can be any of the following – applicationName/ subsystemName/ severity,
    • "constraint": (string)-  the value for the chosen field (if the field is severity, the value should be one: debug, verbose, info, warning, error, critical)

Example request:

{
	"name":"group name",
  "description":"group description",
  "enabled":true,
  "ruleMatchers": [{
		"field": "applicationName",
		"constraint": "<application name>"
		},{
		"field": "subsystemName",
		"constraint": "<subsystem name>"
		},{
		"field": "severity",
		"constraint": "<severity>"
		},{
		"field": "severity",
		"constraint": "<severity>"
		},{
		"field": "severity",
		"constraint": "<severity>"
		}]
}

Delete a Group

The first step is to create a group which will keep our rules.

Method: DELETE
URL: https://api.<domain>/api/v1/external/group/GROUPID
Body: Empty

Rules API

Create a Rule

Method: POST
URL: https://api.<domain>/api/v1/external/rule/GROUPID
Body (the following fields are required/optional):

  • "type": (string)(mandatory) – type of the rule (block, extract, parse, jsonextract, replace, timestampextract, removefields),
  • "description": (string)(optional – default is an empty string) – the description for the rule,
  • "enabled": (boolean)(optional) – indicates rather the rule is active or not when created – if fields is missing or value is empty – default is True,
  • "name": (string)(mandatory) – the name of the rules group,
  • "rule": (string)(mandatory – except in “removefields” and “timestampextract” – for more info look at restrictions) – the regex of the rule (must be a regex that can be compiled),
  • "sourceField": (string)(optional) – If not stated in the body then the default source field will be the log text. If you want to run the rule against any internal log JSON field use text.field_name.field_name2…
  • "destinationField":(string)(optional) – If not stated in the body then the default destination field will be the log text. If you want to run the rule against any internal log JSON field use text.field_name.field_name2…
Rule TypeDescription / Restrictions
BLOCK, EXTRACTAPI request body cannot have: “ReplaceNewVal” OR “DestinationField”
PARSEAPI request body cannot have: “ReplaceNewVal”
JSONEXTRACTAPI request body cannot have: “ReplaceNewVal”. API request body must have: “DestinationField”:(string) – need to be one of: “category”, “className”, “methodName”, “severity”, “threadId”
REPLACEAPI request body must have: “ReplaceNewVal”
REMOVEFIELDSAPI request body must have: “rule”:(string) – comma-separated list of json fields you would like to remove
TIMESTAMPEXTRACTAPI request body cannot have: “rule” or “destinationField” fields. API request body must have:”timeFormat”(string) – you can see our suggestion for each type of standard, normally you will need to change the format to match your time field string pattern with the while comply to the standard “formatStandard”:(string) = need to be one of: “javasdf”, “golang”, “strftime”, “secondsts”, “millits”, “microts”, “nanots”). API request body should contain a meaningful value for “sourceField”: (string) – a value is not specified the default source files will be the text field and the rule will not work.You should set the value as Text.<json_key_with_time_field>)
STRINGIFY JSON FIELDConvert JSON object to JSON string. Find out more here.
PARSE JSON FIELDConvert JSON string to JSON object. Find out more here.

Read a Rule

Method: GET
URL: https://api.<domain>/api/v1/external/rule/RULEID/group/GROUPID
Body: Empty

Update a Rule

Method: PUT
URL: https://api.<domain>/api/v1/external/rule/RULEID/group/GROUPID
Body (the following fields are required/optional):

  • "type": (string)(mandatory) – type of the rule (block, extract, parse, jsonextract, replace, timestampextract, removefields),
  • "description": (string)(optional – default is an empty string) – the description for the rule,
  • "enabled": (boolean)(optional) – indicates rather the rule is active or not when created – if fields is missing or value is empty – default is True,
  • "name": (string)(mandatory) – the name of the rules group,
  • "rule": (string)(mandatory – except in “removefields” and “timestampextract” – for more info look at restrictions) – the regex of the rule (must be a regex that can be compiled),
  • "sourceField": (string)(optional) – If not stated in the body then the default source field will be the log text. If you want to run the rule against any internal log JSON field use text.field_name.field_name2…
  • "destinationField":(string)(optional) – If not stated in the body then the default destination field will be the log text. If you want to run the rule against any internal log JSON field use text.field_name.field_name2…

Note: It is possible to modify only fields which you want to edit but keep in mind that if you do a change which impact other fields then those fields need to be also updated.

Rule TypeDescription / Restrictions
BLOCK, EXTRACTAPI request body cannot have: “ReplaceNewVal” OR “DestinationField”
PARSEAPI request body cannot have: “ReplaceNewVal”
JSONEXTRACTAPI request body cannot have: “ReplaceNewVal”. API request body must have: “DestinationField”:(string) – need to be one of: “category”, “className”, “methodName”, “severity”, “threadId”
REPLACEAPI request body must have: “ReplaceNewVal”
REMOVEFIELDSAPI request body must have: “rule”:(string) – comma-separated list of json fields you would like to remove
TIMESTAMPEXTRACTAPI request body cannot have: “rule” or “destinationField” fields. API request body must have:”timeFormat”(string) – you can see our suggestion for each type of standard, normally you will need to change the format to match your time field string pattern with the while comply to the standard “formatStandard”:(string) = need to be one of: “javasdf”, “golang”, “strftime”, “secondsts”, “millits”, “microts”, “nanots”). API request body should contain a meaningful value for “sourceField”: (string) – a value is not specified the default source files will be the text field and the rule will not work.You should set the value as Text.<json_key_with_time_field>)
STRINGIFY JSON FIELDConvert JSON object to JSON string. Find out more here.
PARSE JSON FIELDConvert JSON string to JSON object. Find out more here.

Example request – disable a rule:

{
    "enabled": false
}

Delete a Rule

Method: DELETE
URL: https://api.<domain>/api/v1/external/rule/RULEID/group/GROUPID

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].

On this page