This guide will help you use our Parsing Rules API to create, read, update or delete parsing rules and rule groups via API.
In order to use the Parsing Rules API, first locate your:
To send an external request, the request headers should contain the following Headers:
Alerts, Rules, and Tags API key can be found under Data Flow –> API Keys.
NOTE – Only users with Admin privileges have access to the Rules API and keys.
This guide will help you use our Rules API to create, read, update or delete parsing rules and rule groups via API. Parsing rules are organized inside Rule Groups. Each group has a name and a set of parsing rules with a logical AND/OR relationship between them.
Logs are processed according to the order of the Rule Groups (top to bottom) and then by the order of rules within the Rule Group and according to the logical operators between them (AND/OR).
Group Name | Sub-Group / Condition | Rule Name |
---|---|---|
My group 1 | My rule 1-1-a | |
OR | ||
My rule 1-1-b | ||
AND | ||
My rule 1-2-a | ||
OR | ||
My rule 1-2-b | ||
My group 2 | My rule 2-1-a |
To create a parsing rule, there is a need to define a rule group and the parsing rule itself:
Method: POST
URL: https://api.<domain>/api/v1/external/rule/rule-set
Body (the following fields are required/optional):
"name"
: (string)(mandatory) – the name of the rules group,"description"
: (string)(optional – default is an empty string) – the description for the rules group,"enabled"
: (boolean)(optional) – indicates rather the group is active or not when created – if fields is missing or value is empty – default is True,"creator"
: (string)(optional – default is “Coralogix external API”) – the name of the group creator,"order"
: (number)(optional) – the rule group order number. If it is not provided, the rule group will be added at the end of the list of rule groups. If the group with the provided order number already exists, new group will be added one below and the order will be changed to +1 automatically,"ruleMatchers"
:(array of objects)(mandatory) – the conditions which the group of rules works on – application/subsystem/severity of logs. Each object in the array should contain:
"field"
: (string) can be any of the following – applicationName/ subsystemName/ severity,"constraint"
: (string)- the value for the chosen field (if the field is severity, the value should be one: debug, verbose, info, warning, error, critical)"rulesGroups"
:(array of objects)(optional) – It contains two fields:
"order"
: (number) – it is the number of the sub-group. If rules are under two different sub-groups then it means there is AND condition between them"rules"
: (array of object) – each object is a ruleThe rules
object keys:
"name"
: (string)(mandatory) – the name of rule,"description"
: (string)(optional – default is an empty string) – the description for the rule,"enabled"
: (boolean)(optional) – indicates rather the rule is active or not when created – if fields is missing or value is empty – default is True,"type"
: (string)(mandatory) – type of the rule – (block, extract, parse, jsonextract, replace, timestampextract, removefields),"sourceField"
: (string)(optional) – If not stated in the body then the default source field will be the log text. If you want to run the rule against any internal log JSON field use Text.field_name.field_name2…"destinationField"
:(string)(optional) – If not stated in the body then the default destination field will be the log text. If you want to run the rule against any internal log JSON field use Text.field_name.field_name2…"order"
: (number)(optional) – if empty then the rule is added at the end"rule"
: (string)(mandatory – except in “removefields” and “timestampextract” – for more info look at restrictions) – the regex of the rule (must be a regex that can be compiled)Rule Type | Description / Restrictions |
---|---|
BLOCK, EXTRACT | API request body cannot have: “ReplaceNewVal” OR “DestinationField” |
PARSE | API request body cannot have: “ReplaceNewVal” |
JSONEXTRACT | API request body cannot have: “ReplaceNewVal”. API request body must have: “DestinationField”:(string) – need to be one of: “category”, “className”, “methodName”, “severity”, “threadId” |
REPLACE | API request body must have: “ReplaceNewVal” |
REMOVEFIELDS | API request body must have: “rule”:(string) – comma-separated list of json fields you would like to remove |
TIMESTAMPEXTRACT | API request body cannot have: “rule” or “destinationField” fields. API request body must have:”timeFormat”(string) – you can see our suggestion for each type of standard, normally you will need to change the format to match your time field string pattern with the while comply to the standard “formatStandard”:(string) = need to be one of: “javasdf”, “golang”, “strftime”, “secondsts”, “millits”, “microts”, “nanots”). API request body should contain a meaningful value for “sourceField”: (string) – a value is not specified the default source files will be the text field and the rule will not work.You should set the value as Text.<json_key_with_time_field>) |
STRINGIFY JSON FIELD | Convert JSON object to JSON string. Find out more here. |
PARSE JSON FIELD | Convert JSON string to JSON object. Find out more here. |
{ "name": "My test group with one rule - parse", "description": "It is an example how to create a group with a parsing rule", "enabled": true, "order": 1, "creator": "[email protected]", "ruleMatchers": [], "rulesGroups": [ { "order": 1, "rules": [ { "name": "Test parse rule", "description": "This is the first parsing rule", "enabled": true, "type": "parse", "sourceField": "text", "destinationField": "text", "order": 1, "rule": "^(?P<timestamp>\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\s*\\\\d{2}:\\\\d{2}:\\\\d{2}\\\\.\\\\d+)\\\\s+(?P<message>.*)$" } ] } ] }
{ "name": "My test group with one rule per each sub-group", "description": "It is an example how to create a group with two rules with AND condition between them ", "enabled": true, "order": 1, "creator": "[email protected]", "ruleMatchers": [], "rulesGroups": [ { "order": 1, "rules": [ { "name": "Test parse rule", "description": "This is the first parsing rule", "enabled": true, "type": "parse", "sourceField": "text", "destinationField": "text", "order": 1, "rule": "^(?P<timestamp>\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\s*\\\\d{2}:\\\\d{2}:\\\\d{2}\\\\.\\\\d+)\\\\s+(?P<message>.*)$" } ] }, { "order": 2, "rules": [ { "name": "Test replace rule", "description": "This is the first replace rule", "enabled": true, "type": "replace", "sourceField": "text.message", "destinationField": "text.message", "order": 1, "rule": "host", "replaceNewVal": "hostname" } ] } ] }
{ "name": "My test group with three rules", "description": "It is an example how to create a group with 3 rules", "enabled": true, "order": 5, "creator": "[email protected]", "ruleMatchers": [], "rulesGroups": [ { "order": 1, "rules": [ { "name": "Test parse rule", "description": "This is the first parsing rule", "enabled": true, "type": "parse", "sourceField": "text", "destinationField": "text", "order": 1, "rule": "^(?P<timestamp>\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\s*\\\\d{2}:\\\\d{2}:\\\\d{2}\\\\.\\\\d+)\\\\s+(?P<message>.*)$" } ] }, { "order": 2, "rules": [ { "name": "Test replace rule 1", "description": "This is the first replace rule", "enabled": true, "type": "replace", "sourceField": "text.message", "destinationField": "text.message", "order": 1, "rule": "host", "replaceNewVal": "hostname" }, { "name": "Test replace rule 2", "description": "This is the second replace rule", "enabled": true, "type": "replace", "sourceField": "text.message", "destinationField": "text.message", "order": 2, "rule": "server", "replaceNewVal": "hostname" } ] } ] }
To get all parsing rules:
Method: GET
URL: https://api.<domain>/api/v1/external/rules
In response, you will get the "companyRulesData"
array with groups and rules.
{ "companyRulesData": [ { "id": "20538afe-12d4-4032-a2d9-42b14e0036a1", "name": "My test group with one rule - parse", "description": "It is an example how to create a group with a parsing rule", "order": 1, "enabled": true, "creator": "[email protected]", "updatedAt": "2022-11-03 08:32:53", "createdAt": "2022-11-03 08:32:53", "ruleMatchers": null, "rulesGroups": [ { "id": "562cbdf3-5021-42ab-a85a-23c0e80db62b", "order": 1, "rules": [ { "id": "f3ddb109-c80d-4a46-8aa9-98f2984fb2f7", "name": "Test parse rule", "description": "This is the first parsing rule", "enabled": true, "ruleMatchers": null, "rule": "^(?P<timestamp>\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\s*\\\\d{2}:\\\\d{2}:\\\\d{2}\\\\.\\\\d+)\\\\s+(?P<message>.*)$", "sourceField": "text", "destinationField": "text", "replaceNewVal": "", "type": "parse", "order": 1, "keepBlockedLogs": false, "deleteSource": false, "escapedValue": false, "overrideDest": false } ] } ], "hidden": false }, { "id": "94b0a95b-994e-42ef-9c22-48d58b2e339e", "name": "My test group with one rule per each sub-group", "description": "It is an example how to create a group with two rules with AND condition between them ", "order": 2, "enabled": true, "creator": "[email protected]", "updatedAt": "2022-11-03 08:52:51", "createdAt": "2022-11-03 08:52:39", "ruleMatchers": null, "rulesGroups": [ { "id": "4dfde3e7-2791-4b34-aa0a-3edafe719ddb", "order": 1, "rules": [ { "id": "5802ec65-dae5-4e20-bd1d-37dbf15e2630", "name": "Test parse rule", "description": "This is the first parsing rule", "enabled": true, "ruleMatchers": null, "rule": "^(?P<timestamp>\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\s*\\\\d{2}:\\\\d{2}:\\\\d{2}\\\\.\\\\d+)\\\\s+(?P<message>.*)$", "sourceField": "text", "destinationField": "text", "replaceNewVal": "", "type": "parse", "order": 1, "keepBlockedLogs": false, "deleteSource": false, "escapedValue": false, "overrideDest": false } ] }, { "id": "ca69375b-a869-4c6f-96eb-14a4e6c8a935", "order": 2, "rules": [ { "id": "62e46153-a56a-4322-a1be-c4db871afaec", "name": "Test replace rule", "description": "This is the first replace rule", "enabled": true, "ruleMatchers": null, "rule": "host", "sourceField": "text.message", "destinationField": "text.message", "replaceNewVal": "hostname", "type": "replace", "order": 1, "keepBlockedLogs": false, "deleteSource": false, "escapedValue": false, "overrideDest": false } ] } ], "hidden": false } ] }
As you can see, each group contains the id
field which is the GROUPID.
The GROUPID can be also obtained from the Parsing Rules page.
Query for parsing rules from the specific group.
Method: GET
URL: https://api.<domain>/api/v1/external/rule/rule-set/<GROUPID>
https://api.coralogix.com/api/v1/external/rule/rule-set/20538afe-12d4-4032-a2d9-42b14e0036a1
{ "id": "20538afe-12d4-4032-a2d9-42b14e0036a1", "name": "My test group with one rule - parse", "description": "It is an example how to create a group with a parsing rule", "order": 1, "enabled": true, "creator": "[email protected]", "updatedAt": "2022-11-03 08:32:53", "createdAt": "2022-11-03 08:32:53", "ruleMatchers": null, "rulesGroups": [ { "id": "562cbdf3-5021-42ab-a85a-23c0e80db62b", "name": "", "order": 1, "enabled": true, "rules": [ { "id": "f3ddb109-c80d-4a46-8aa9-98f2984fb2f7", "name": "Test parse rule", "description": "This is the first parsing rule", "enabled": true, "ruleMatchers": null, "rule": "^(?P<timestamp>\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\s*\\\\d{2}:\\\\d{2}:\\\\d{2}\\\\.\\\\d+)\\\\s+(?P<message>.*)$", "sourceField": "text", "destinationField": "text", "replaceNewVal": "", "type": "parse", "order": 1, "keepBlockedLogs": false, "deleteSource": false, "escapedValue": false, "overrideDest": false } ], "type": "user" } ] }
Modify or update a group with one parsing rule.
Method: PUT
URL: https://api.<domain>/api/v1/external/rule/rule-set
To update any rule just perform GET with a GROUPID, modify all rules in the group from the response, and send it with PUT request.
Note: The parameter ruleMatchers at the group section is returned by the GET request. It needs to be changed to the empty array [] before sending the request.
Method: DELETE
URL: https://api.<domain>/api/v1/external/rule/rule-set/<GROUPID>
Method: DELETE
URL: https://api.<domain>/api/v1/external/rule/RULEID/group/GROUPID
where RULEID is the id
of the rule.
Note: It is not possible to delete a sub-group of rules.
If there is a need to copy all rules from one team to another follow these steps:
NOTE: When moving rules from one account to another keep in mind that the rule groups you imported are concatenated after the existing rules groups, we do not override existing groups by default. If you want to override the existing groups you should first remove them entirely before importing new rule groups.
The REST API methods below are the old ways to manage parsing rules. They do not support the order of rules and fields names may be different. They are still supported.
Create a group which will keep your rules.
Method: POST
URL: https://api.<domain>/api/v1/external/group
Body (the following fields are required/optional):
"name"
: (string)(mandatory) – the name of the rules group,"description"
: (string)(optional – default is an empty string) – the description for the rules group,"enabled"
: (boolean)(optional) – indicates rather the group is active or not when created – if fields is missing or value is empty – default is True,"creator"
: (string)(optional – default is “Coralogix external API”) – the name of the group creator,"ruleMatchers"
:(array of objects)(mandatory) – the conditions which the group of rules work on – application/subsystem/severity of logs. Each object in the array should contain:
"field"
: (string) can be any of the following – applicationName/ subsystemName/ severity,"constraint"
: (string)- the value for the chosen field (if the field is severity, the value should be one: debug, verbose, info, warning, error, critical)Example request:
{ "name": "My group 1", "description": "This is a test group", "enabled": true, "creator": "[email protected]", "ruleMatchers": [{ "field": "applicationName", "constraint": "test_app" }, { "field": "subsystemName", "constraint": "test_sub" }, { "field": "severity", "constraint": "warning" }, { "field": "severity", "constraint": "error" }, { "field": "severity", "constraint": "critical" }] }
Example response:
The rule was created at the bottom of all rule groups.
{ "id": "73cac31e-225b-41d8-bc32-5a825d2b29a1", "name": "My group 1", "description": "This is a test group", "order": 20, "enabled": true, "creator": "[email protected]", "createdAt": "2022-11-02 14:47:56", "ruleMatchers": [ { "field": "applicationName", "constraint": "test_app" }, { "field": "subsystemName", "constraint": "test_sub" }, { "field": "severity", "constraint": "warning" }, { "field": "severity", "constraint": "error" }, { "field": "severity", "constraint": "critical" } ], "rulesGroups": [] }
The first step is to create a group which will keep our rules.
Method: GET
URL: https://api.<domain>/api/v1/external/group/GROUPID
Body: Empty
The first step is to create a group that will keep our rules.
Method: PUT
URL: https://api.<domain>/api/v1/external/group/GROUPID
Body (the following fields are required/optional):
"name"
: (string)(mandatory) – the name of the rules group,"Description"
: (string)(optional – default is an empty string) – the description for the rules group,"enabled"
: (boolean)(optional) – indicates rather the group will be set to active or not when updating,"ruleMatchers"
:(array of objects)(mandatory) – the conditions which the group of rules work on – application/subsystem/severity of logs. Each object in the array should contain:
"field"
: (string) can be any of the following – applicationName/ subsystemName/ severity,"constraint"
: (string)- the value for the chosen field (if the field is severity, the value should be one: debug, verbose, info, warning, error, critical)Example request:
{ "name":"group name", "description":"group description", "enabled":true, "ruleMatchers": [{ "field": "applicationName", "constraint": "<application name>" },{ "field": "subsystemName", "constraint": "<subsystem name>" },{ "field": "severity", "constraint": "<severity>" },{ "field": "severity", "constraint": "<severity>" },{ "field": "severity", "constraint": "<severity>" }] }
The first step is to create a group which will keep our rules.
Method: DELETE
URL: https://api.<domain>/api/v1/external/group/GROUPID
Body: Empty
Method: POST
URL: https://api.<domain>/api/v1/external/rule/GROUPID
Body (the following fields are required/optional):
"type"
: (string)(mandatory) – type of the rule – (block, extract, parse, jsonextract, replace, timestampextract, removefields),"description"
: (string)(optional – default is an empty string) – the description for the rule,"enabled"
: (boolean)(optional) – indicates rather the rule is active or not when created – if fields is missing or value is empty – default is True,"name"
: (string)(mandatory) – the name of the rules group,"rule"
: (string)(mandatory – except in “removefields” and “timestampextract” – for more info look at restrictions) – the regex of the rule (must be a regex that can be compiled),"sourceField"
: (string)(optional) – If not stated in the body then the default source field will be the log text. If you want to run the rule against any internal log JSON field use text.field_name.field_name2…"destinationField"
:(string)(optional) – If not stated in the body then the default destination field will be the log text. If you want to run the rule against any internal log JSON field use text.field_name.field_name2…Rule Type | Description / Restrictions |
---|---|
BLOCK, EXTRACT | API request body cannot have: “ReplaceNewVal” OR “DestinationField” |
PARSE | API request body cannot have: “ReplaceNewVal” |
JSONEXTRACT | API request body cannot have: “ReplaceNewVal”. API request body must have: “DestinationField”:(string) – need to be one of: “category”, “className”, “methodName”, “severity”, “threadId” |
REPLACE | API request body must have: “ReplaceNewVal” |
REMOVEFIELDS | API request body must have: “rule”:(string) – comma-separated list of json fields you would like to remove |
TIMESTAMPEXTRACT | API request body cannot have: “rule” or “destinationField” fields. API request body must have:”timeFormat”(string) – you can see our suggestion for each type of standard, normally you will need to change the format to match your time field string pattern with the while comply to the standard “formatStandard”:(string) = need to be one of: “javasdf”, “golang”, “strftime”, “secondsts”, “millits”, “microts”, “nanots”). API request body should contain a meaningful value for “sourceField”: (string) – a value is not specified the default source files will be the text field and the rule will not work.You should set the value as Text.<json_key_with_time_field>) |
STRINGIFY JSON FIELD | Convert JSON object to JSON string. Find out more here. |
PARSE JSON FIELD | Convert JSON string to JSON object. Find out more here. |
Method: GET
URL: https://api.<domain>/api/v1/external/rule/RULEID/group/GROUPID
Body: Empty
Method: PUT
URL: https://api.<domain>/api/v1/external/rule/RULEID/group/GROUPID
Body (the following fields are required/optional):
"type"
: (string)(mandatory) – type of the rule – (block, extract, parse, jsonextract, replace, timestampextract, removefields),"description"
: (string)(optional – default is an empty string) – the description for the rule,"enabled"
: (boolean)(optional) – indicates rather the rule is active or not when created – if fields is missing or value is empty – default is True,"name"
: (string)(mandatory) – the name of the rules group,"rule"
: (string)(mandatory – except in “removefields” and “timestampextract” – for more info look at restrictions) – the regex of the rule (must be a regex that can be compiled),"sourceField"
: (string)(optional) – If not stated in the body then the default source field will be the log text. If you want to run the rule against any internal log JSON field use text.field_name.field_name2…"destinationField"
:(string)(optional) – If not stated in the body then the default destination field will be the log text. If you want to run the rule against any internal log JSON field use text.field_name.field_name2…Note: It is possible to modify only fields which you want to edit but keep in mind that if you do a change which impact other fields then those fields need to be also updated.
Rule Type | Description / Restrictions |
---|---|
BLOCK, EXTRACT | API request body cannot have: “ReplaceNewVal” OR “DestinationField” |
PARSE | API request body cannot have: “ReplaceNewVal” |
JSONEXTRACT | API request body cannot have: “ReplaceNewVal”. API request body must have: “DestinationField”:(string) – need to be one of: “category”, “className”, “methodName”, “severity”, “threadId” |
REPLACE | API request body must have: “ReplaceNewVal” |
REMOVEFIELDS | API request body must have: “rule”:(string) – comma-separated list of json fields you would like to remove |
TIMESTAMPEXTRACT | API request body cannot have: “rule” or “destinationField” fields. API request body must have:”timeFormat”(string) – you can see our suggestion for each type of standard, normally you will need to change the format to match your time field string pattern with the while comply to the standard “formatStandard”:(string) = need to be one of: “javasdf”, “golang”, “strftime”, “secondsts”, “millits”, “microts”, “nanots”). API request body should contain a meaningful value for “sourceField”: (string) – a value is not specified the default source files will be the text field and the rule will not work.You should set the value as Text.<json_key_with_time_field>) |
STRINGIFY JSON FIELD | Convert JSON object to JSON string. Find out more here. |
PARSE JSON FIELD | Convert JSON string to JSON object. Find out more here. |
Example request – disable a rule:
{ "enabled": false }
Method: DELETE
URL: https://api.<domain>/api/v1/external/rule/RULEID/group/GROUPID
Need help?
Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.
Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].