RegEx 101: Guide for Managing Log Data in Coralogix

You may be wondering, what are RegEx expressions? And how do they work when it comes to managing log data in Coralogix? We’ve got you covered.

Regular expressions can be crucial for wrangling log data efficiently. You may want to extract specific data from your logs to make it easier to analyze and visualize. Sometimes you might want to capture an email when a particular message is logged. Other times, you may need to hide sensitive data in logs before they are saved.

And more often than not, you need to match using a RegEx pattern rather than an exact text search.

This RegEx guide is split into three parts. We’ll start with a brief intro on understanding RegEx, while the second part shows RegEx methods and how you can use them in a logging context. Finally, we end with a complete RegEx 101 table for your quick reference

What is RegEx and how does it work?

Regular Expressions, also known as RegEx or RegExp, is a domain-specific language (DSL) used for pattern searches and replacements.

There are times when you may not need a regular expression; you can just search for a specific text. For example, if you just want to find log lines with the text “user logged in”, you can simply enter this text into your log search tool. But if your log lines look like this: “user_32 logged in” you can’t search for the exact text anymore since the user id is a variable that changes.

Fortunately, there is a RegEx pattern for this case which makes use of the match anything sequence:

user_d+ logged in

How to use RegEx to manage log data

In this section, we list all places where RegEx may be used in Coralogix and give you some ready-made examples which you can start using immediately.

Extracting text from logs

One of the most wanted things is extracting data from logs. For example, this way, you can convert unstructured logs into structured logs. For this, you’ll need to use the Extract Rule.

Extracting text into Custom JSON Fields

Suppose you have an unstructured log of the following form:

`${logLevel}: World-${worldName}: ${logText}`

and you would like to convert all entries of this format into a JSON object of the following form:

{
    "level": `${log-level}`,
    "tag": `World-${worldName}`,
    "text": `${logText}`
}

The Extract Rule will extract all text matched by capturing groups and convert it into a JSON object. Names of the capturing groups will become JSON keys.

For example:

^(?P.*?):s*(?P.*?):

Some explanations:

• ^ is the start of line symbol, it means that match should start with the start of a line
• (?PX), where X is the matcher, is the named capturing group syntax. We have three capturing groups here, one for each JSON key we need.
• s means “any whitespace character“
• . means “any one character”
• * means “0 or more matches of the previous symbol/character”
• .* means “1 or more matches of the previous symbol/character”
• .*? means “any characters any number of times, with the least amount of tokens necessary“

Thus, the RegEx above means:

• match any text starting with the start of a line and until the first : symbol as capturing group "level"
• then, after any number of whitespaces, match any text until the next : symbol as capturing group "tag"

The whole log text is automatically set to the "text" field by Coralogix, so we will have the "text" field automatically.

Above RegEx will convert the following log:

"info: World-w-8: generate: new world"

into this JSON object:

{
   "level": "info",
   "tag": "World-w-8",
   "text": "info: World-w-8: generate: new world"
}

Extracting text into Predefined Coralogix Fields

But you don’t always want to extract text into custom JSON fields. Sometimes you need to assign matched text to predefined Coralogix fields. In the example above:

"info: World-w-8: generate: new world"

You might want to extract the "info" text into the Severity column, and the "World" text into the Class column.
You only need to set correct names of the capturing groups for that:

^(?P.*?):s*(?P.*?)-(?P.*?):

This RegEx will format the above message in the following way:

You can set other predefined columns like this, such as Category and Method. You need to set the keys "category" and "methodName" for this. Here is a RegEx mapping these predefined fields:

^(?P.*?):s*(?P.*?)-(?P.*?):s*(?P.*?)[$:]

And here is the result:

Extracting specific data from Logs

We already saw how you can extract data from unstructured logs into custom JSON fields. The same trick can be used for the structured logs too. Following is an example showing you how to extract data from a specific JSON filed.

Suppose you have a structured log line like this:

{
    "type": `${text}`,
    "log": `${text}`,
    "region": "rg-europe-2"
}

Now, knowing that the "region" field has the following form:

`rg-${"europe"|"asia"|"na"}-${number}`

We want to extract the part which tells us whether the region is "europe", "asia" or "na". The RegEx should be like this:

"region"s*:s*"rg-(?P.*?)-

Note the named capturing group "regionName" here – this is what actually extracts text. We place it after the key name "region" and the characters "rg-": right where we expect it to be according to our format. The purpose of the s* symbols is to make RegEx still work if there are any whitespaces before or after the : symbol.

Here is the result of applying this rule:

Replacing and removing values

One of the most common examples where we need to replace or remove values is hiding personal data. Suppose you log phone numbers somewhere and don’t want those to be saved in Coralogix.

We should start with creating a new Rule, as described above. Once we select the Replace rule in the rule type selection section, we will see the following screen:

Suppose we have an unstructured log line like this:

"info: Sender: sendSms: sending sms to phone number +12345678910 to user Andrew"

You want to remove phone number and name from this line. This RegEx will match the line above, starting with "sending sms":

sending sms to phone number +*d+ to user .*

Note that we need to escape the + symbol with because it has a special meaning in the RegEx syntax. This meaning is “1 or more previous characters”. Symbol d matches any single digit. Remember, that the * symbol means “0 or more previous characters”. Thus, +*d+ matches one or more digits which may be prepended by a + symbol (or not).

This RegEx will replace the text matched by the previous RegEx to the same text but without the phone number and name:

sending sms to phone number * to user *

And here is the result of applying the above rule:

Replacing JSON Values in structured logs

Replacing JSON values in structured logs is similar. The whole JSON string is used as the input for the Replace rule.

Let’s remember our structured log line from above:

{
   "type": `${text}`,
   "log": `${text}`,
   "region": "rg-europe-2"
}

Suppose you need to replace the value of the "type" field with any other value you want. Here is how you can match the value of the "type" field:

"type"s*:s*".*?"

Remember that we need the s* symbols to make sure the RegEx works if there are whitespaces before or after the : character.

And here is the RegEx to replace any value of the "type" field with "newType":

"type":"newType"

Using backreferences

Suppose we need to replace “europe” to “eu” in the strings of format “west-europe-2”. And suppose we need to do this not only in the “region” field but in any other part of the log where we meet it. Matching this pattern is rather easy:

.+?-europe-d+

But replacing it using the methods we used before might be rather hard. This is because we need to insert two strings before and after the “europe” text, which might vary. To do this, we first need to capture this text into capturing groups:

(.+?)-europe-(d+) 

Remember that the d symbol means “any digit” and together with + it means “any digit one or more times”. In a similar way, .+ means any symbol one or more times, but with the fewest amount of tokens to make the match.

This will capture the text before “europe” as capturing group 1, and the text after “europe” as capturing group 2. The following RegEx will use backreferences to insert matched content of those groups:

$1-eu-$2

Logs before applying our rule:

Logs after applying our rule:

Searching for logs

Another place where you can use RegEx is the Logs search bar:

You can enter the exact text you want to find there, without any RegEx. But if you look for a particular pattern, you need to use RegEx. RegEx queries have their own form, which is:

/${fieldName}.keyword:/REGEX//

Suppose we have many JSON-structured logs of the following form:

{    
    "log":  `${text}` ,
    "regionName":  `${text}`,
    "region":  `${text}`, 
    "type"  :  `ltest-w-${number}`
}

And we want to match only those entries where "type" is equal to “ltest-w-1”, “ltest-w-2” or “ltest-w-3”. The following search query will do this:

/type.keyword:/ltest-w-[1-3]//

Note: the text between square brackets [ and ], is called a character class. It matches any one character listed between square brackets. Dash may be used as a shorthand to list several characters: [1-5] is the same as [12345]

You can use the same syntax to match data in unstrucutred logs with RegEx. You only need to set field name to text.keyword. For example:

 /text.keyword:/.*ltest-w-[1-3].*// 

Above RegEx will search for text “ltest-w-1”, “ltest-w-2” or “ltest-w-3” in main log body.

Similar RegEx syntax is used in the Kibana search bar. Click Kibana and then Discover to see this search input:

The only difference here is that you don’t need the / symbols in the beginning and at the end. Thus, our search query above will turn into:

type.keyword:/ltest-w-[1-3]/

Note: sometimes you don’t see any logs after clicking Discover. It’s possible that there are no logs for the time interval set by default, which is 15 minutes. Try clicking the time interval selector and increasing it.

Triggering alerts with RegEx

Another popular use case for RegEx in Coralogix is Alerts. Alerts syntax is the same as logs query syntax. Let’s start with creating an alert. First, click the Alerts tab, and then click the Plus button on the right:

Let’s say we want to alert us on a line of the following form:

`App: init: World-${name}: generation error: ${err}`

And suppose we want to get alerts only for worlds "w-1", "w-2", "w-3" or "w-4". Our alert RegEx will look like this:

/text.keyword:/.*World-w-[1-4]: generation error.*//

Remember, that [1-4] matches any single character from 1 to 4, and .* matches any characters any number of times.

In the “Rules definitions” section select “Notify immediately” to get alert on the first match. Or you can select More and then choose the number of matches needed to trigger the alert.

In the Notifications definition section, enter email addresses where a message should be sent on alert.

When an alert is triggered, you will:

• Get an email
• Will see it in the dashboard
• Will see it in the alerts section

Summary

I hope you found this tutorial interesting and helpful. You can start with Regular Expressions by copy-pasting the examples above and substituting parts of them when you need it. You will soon find that knowing RegEx makes your life easier not just with Coralogix, but with many other products.

About the author
Andrei Chernikov is a Full-Stack Developer and also writes helpful guides on Medium.

(This article was updated in August 2023)