Tag: parsing

June 2022 Platform Updates

Posted on by eugene evdokimov

Our team has been hard at work this month to introduce 2 new parsing rules, DataMap improvements, updated tracing visualizations for SLA monitoring & more.

Get up to speed on everything that’s new and improved in the Coralogix platform!

New Parsing Rules

Learn More >

This month we are introducing 2 new parsing rules to bring more value to customers who have many fields and nested fields in their log data.

parsing newsletter2

The new Stringify JSON Field and Parse JSON Field rules enable you to parse escaped JSON values within a field to a valid JSON object and vice versa – stringify a JSON object to an escaped string.

Learn More >

DataMap Updates

Learn More >

The DataMap allows you to build custom mappings of your infrastructure using metric data for monitoring system health and quickly identifying issues.

datamap newsletter2

In the Group Editor, you’ll find new options to:

  • Sort the display by attributes (e.g. sort by severity for defined thresholds)
  • Scale threshold values to make metric graphs more readable
  • Limit the number of hexagons shown per group

In the DataMap display, use new ‘Compare to others’ functionality to compare an element with 10 others in the same group. Plus, expand and collapse specific groups to minimize the number of displayed elements.

Learn More >

Tracing Updates

Learn More >

New dynamic graphs and saved views in the Tracing UI enable it to serve as SLA dashboards for any application or service.

tracing newsletter2

In addition to the original default graph for Max duration by Action, there are now two additional default graphs for Count by Service and Error Count by Service.

All three graphs can be customized, and aggregation operators have been added for 99, 95, and 50th percentiles to help deepen your ability to monitor business SLOs.

When investigating traces in the explore section, you can now save your current view and load saved views just like you do in the Logs UI.

Learn More >

*Note that the aggregation operators, as well as the Duration filter in the sidebar, are run over the Spans.

Archive Query Updates

Learn More >

Improvements to the archive query now allow timeframes up to 3 days for added accessibility to data in your remote bucket.

archive query newsletter2

Additional updates to the Archive Query in Explore Screen include:

  • New Execute Archive Query function allows you to review active filters before clicking ‘Run Query’. To prevent unexpected wait times, queries will no longer run automatically when switching from Logs to Archive. 
  • Non-optimal archive queries (e.g. “hello”) will trigger a warning pop up recommending to improve the query conditions.

Learn More >

New Integrations

Amazon Kinesis Data Firehose

Stream large volumes of logs and metrics to Coralogix and reduce operational overhead using our integration with Amazon Kinesis Data Firehose.

Learn More >

Terraform Modules

Easily install and manage Coralogix integrations using Terraform modules in your infrastructure code.

Google Cloud Pub/Sub

Use our predefined function to forward your logs from Google Cloud’s Pub/Sub straight to Coralogix.

Learn More >

GitHub Version Tags

Use the cURL command in GitHub Actions to insert a new tag when a release is published or when a pull request is closed.

Learn More >

Coralogix RabbitMQ Agent

Pull metrics from RabbitMQ Admin UI and send them to your Coralogix account using our AWS Lambda function.

Learn More >

Salesforce Cloud Commerce

Ingest security logs from Salesforce for admin and compliance monitoring.

Learn More >

How JS Works Behind The Scenes — The Engine

Posted on by eugene evdokimov

Have you ever asked yourself “how does this all work behind the scenes?”. I know I have.

Having a deep understanding of certain concepts allows us to understand code in a much better way, perform better in our job and in addition to that it’s super helpful in job interviews.

And also it can be a super fun subject to learn… So with that being said, let’s get into the details of how the JS engine works.

In this post we will dive deep into the world of JS, how it works behind the scenes, from the Engine, to concepts like hoisting, execution context, lexical environment and more.

Here’s a general overview of how a JS engine works:

js overview

Don’t worry if you don’t understand any of this yet, at the end of the article you’ll understand every step of this diagram.

Let’s go!

Environment

A computer, a compiler, or even a browser can’t actually ‘understand’ code that’s written in JS. If so, how does the code run?

Behind the scenes, JS always runs in a certain environment, most common ones are:

  1. Browser (by far the most common)
  2. Node.js (which is a runtime environment which allows you to run JS outside of the browser, usually in servers)

Engine

js engine

So JS needs to run in a certain environment, but what exactly is in the environment?

When you write code in JS, you write it in human-readable syntax, with alphabets and numbers. As mentioned, a machine can not understand this type of code.

This is why each environment has an engine.

In general, the engine’s job is to take that code and transform it into machine code which can eventually be run by the computer processor.

Each environment has its own engine, the most common ones are Chrome V8 (which Node also uses), Firefox SpiderMonkey, JavaScriptCore by Safari and Chakra by IE.

All engines work in a similar fashion but there are differences between each engine.

It’s also important to keep in mind that behind the scenes an engine is simply a software, Chrome V8 for example is a software written in C++.

Parser

js parser

So we have an environment, and we have an engine inside that environment. The first thing the engine does upon executing your code is check the code using the parser.

The parser knows JS syntax and rules, and its job is to go through the code line by line and check if the syntax of the code is correct.

If the parser comes across an error, it stops running and sends out an error. If the code is valid, the parser generates something that’s called an Abstract Syntax Tree (or AST for short).

Abstract Syntax Tree (AST)

abstract syntax tree

So our environment has an engine, which has a parser, which generates an AST. But what is an AST and why do we need it?

AST is a data structure, which is not unique to JS but actually used by a lot of other languages (some of them are Java, C#, Ruby, Python).

An AST is simply a tree representation of your code, and the main reason the engine creates an AST instead of compiling directly to a machine code is that it’s easier to convert to machine code when you have the code inside a tree data structure.

You can actually check out how the AST looks like, just put any code in the website ASTExplorer and check out the data structure that’s created:

astexplorer

The Interpreter

js interpreter

The Interpreter’s job is to take the AST that has been created and transform it into an Intermediate Representation of the code (IR).

We will learn more about the interpreter later on, as further context is required in order to fully understand what it is.

The Intermediate Representation (IR)

So what is this IR the interpreter generates from the AST?

An IR is a data structure or code which represents the source code. Its role is to be an intermediate step between code that’s written in an abstract language such as JS and machine code.

Essentially you can think of IR as an abstraction of machine code.

There are many types of IR, a very popular among JS engines is Bytecode. Here’s a picture which demonstrates the IR role in the V8 engine:

intermediate representation IR

But you might be asking… Why do we need to have an IR? Why not just compile straight to machine code. There are 2 primary reasons why Engines use IR as an intermediate step between high-level code and machine code:

  1. Mobility — when code gets compiled to machine code, it needs to match the hardware that it’s run on.

Machine code written for an Intel processor and machine code written for an ARM processor will be different. An IR, on the other-hand, is universal and can match any platform. This makes the following conversion process easier and more mobile.

  1. Optimizations — it’s easier to run optimizations with IR compared to machine code, this is true both from code optimizations point of view and hardware optimizations.

Fun fact: JS engines are not the only ones using Bytecode as an IR, among the languages which also use Bytecode you will find C#, Ruby, Java, and more.

The Compiler

js compiler

The compiler’s job is to take the IR which the interpreter created, which is in our case Bytecode, and transform it into a machine code with certain optimizations.

Let’s talk about code compilation and some fundamental concepts. Keep in mind that this is a huge subject that takes a lot of time to master, so I’ll only touch on it generally for our use case.

Interpreters vs Compilers

There are two ways to translate code into something that a machine can run, using a compiler and using an interpreter.

The difference between an interpreter and a compiler is that an interpreter translates your code and executes it line-by-line while a compiler instantly translates all code into machine code before executing it.

There are pros and cons to each, a compiler is fast but complex and slow to start, an interpreter is slower but simpler.

With that being said, there are 3 ways to turn high-level code into machine code and run it:

  1. Interpretation — with this strategy you have an Interpreter which goes through the code line by line and executes it (not so efficient).
  2. Ahead of Time Compilation (AOT) — here you have a compiler first compiling the entire code, and only then executing it.
  3. Just-In-Time Compilation — Combination between the AOT strategy and the interpretation strategy, a JIT compilation strategy attempts to take the best from both worlds, performing dynamic compilation, but also allowing certain optimizations to happen, which really speeds up the compilation process. We’ll explain more about JIT compilation.

Most JS engines use a JIT compiler but not all of them. For example Hermes, the engine which React Native uses, doesn’t use a JIT compiler.

To summarize, in JS Engines the compiler takes the IR created by the interpreter and generates optimized machine code from it.

JIT Compiler

Like we said, most JS Engines use a JIT compilation method. The JIT combines both the AOT strategy and interpretation, allowing for certain optimizations to happen. Let’s dive deeper into these optimizations and what exactly the compiler does.

JIT compilation optimizations are done by taking code that’s repeating itself and optimizing it. The optimizations process works as follows:

In essence, a JIT compiler gets feedback by collecting profiling data for the code that’s executed, if it comes across any hot code segment (code which repeats itself), the hot segment will go through the compiler which will then use this information to re-compile more optimally. 

Let’s say you have a function, which returns a property of an object:

function load(obj) {
return obj.x;

}

Looks simple? Maybe to us, but for the compiler this is not a simple task. If the compiler sees an object it knows nothing about, it has to check where is the property x, if the object indeed has such a property, where is it in the memory, is it in the prototype chain and much more.

So what does it do to optimize it?

In order to understand that, we must know that in machine code, the object is saved with its types. 

Let’s assume we have an object with x and y properties, the x is of type number and the y is of type string. Theoretically, the object will be represented in machine code like this:

obj:

x: number

y: string

Optimization can be done if we call a function with the same object structure. This means that properties will be the same and in the same order, but values can be different, like this:

load({x: 1, y: 'hello'});

load({x: 5, y: 'world'});

load({x: 3, y: 'foo'});

load({x: 9, y: 'bar'});

Here’s how it works. Once we call that function, the optimized compiler will recognize we are trying to call a function that’s already been called again.

It will then proceed to check whether the object that’s being passed as an argument has the same properties. 

If so, it will already be able to access its location in memory, instead of looking through the prototype chain and doing many other things that are done for unknown objects.

Essentially the compiler runs through a process of optimization and de-optimization. 

When we run code, the compiler assumes that a function will use the same types it used before, so it saves the code with the types in advance. This type of code is called optimized machine code.

Every time the code calls the same function again, the optimized compiler will then try to access the same place in memory. 

But because JS is a dynamically-typed language, at some point we might want to use the same function with different types. In such a case the compiler will do a process of de-optimization, and compile the code normally.

To summarize the part about the JIT compiler, the JIT compiler’s job is to improve performance by using hot code segments, when the compiler executes code that’s been executed before, it assumes that the types are the same and uses the optimized code that’s been generated. If the types are different the JIT performs a de-optimization and compiles the code normally.

A Note About Performance

One way to improve the performance of your app is to use the same types with different objects. If you have two different objects with the same type, even though the values are different as long as the properties are in the same order and have the same type, the compiler sees these two objects as an object with an equal structure and types and it can access it faster.

For example:

const obj = {
 x: 1,
 a: true,
 b: 'hey'

}

const obj2 = {
 x: 7,
 a: false,
 b: 'hello'

}

As you can see in the example, we have two different objects with different values, but because the properties order and types are the same, the compiler will be able to compile these objects faster.

Although it’s possible to optimize code this way, my opinion is that there are much more important things to do for performance, and something as minor as this shouldn’t concern you.

It’s also hard to enforce something like this in a team, and overall doesn’t seem to make a big difference as the engine is very fast.

With that being said I’ve seen this tip being recommended by a V8 team member, so maybe you do want to try to follow it sometimes. I see no harm in following it when possible, but definitely not at the cost of clean code and architectural decisions.

Summary

  1. JS code has to run in an environment, the most common ones are browsers and Node.js.
  2. The environment needs to have an engine, which takes the JS code that’s written in human-readable syntax and turns it into machine code.
  3. The engine uses a parser to go through the code line by line and check if the syntax is correct. If there are any errors, code will stop executing and an error will be thrown.
  4. If all checks pass, the parser creates a tree data structure called an Abstract Syntax Tree (AST).
  5. The AST is a data structure which represents the code in a tree like structure. It’s easier to turn code into machine code from an AST.
  6. The interpreter then proceeds to take the AST and turn it into IR, which is an abstraction of machine code and an intermediary between JS code and machine code. IR also allows to perform optimizations and is more mobile.
  7. The JIT compiler then takes the IR generated and turns it into machine code, by compiling the code, getting feedback on the fly and using that feedback to improve the compilation process.

This post was originally published at borderlessengineer.com

Instantly Parse The Top 12 Log Types with Coralogix

Posted on by eugene evdokimov

Throughout the past few months, I had the opportunity to work with and serve hundreds of Coralogix’s customers, the challenges in performing efficient Log Analytics are numerous, from log monitoring, collecting, searching, visualizing, and alerting. What I have come to learn is that at the heart of each and every one of these challenges laid the challenge of data parsing. JSON structured logs are easier to read, easier to search, alert, and visualize. They can be queried using the ES API’s, exported to Excel sheets, and even be displayed in Grafana.  So why is it that a lot of logs are still plain text by default and not structured?

As our focus here in Coralogix was always about our customers and their needs, we developed a parsing engine that allows a single UI to parse, extract, mask, and even exclude log entries in-app, or via API. To get you started with log parsing,  we created pre-defined parsing rules for the 12 most common logs on the web.

In this post, we collected the following log templates and created their own Named group REGEX in order to parse them into JSON structure logs in Coralogix: Apache logs, IIS, logs, MongoDB logs, ELB logs, ALB logs, CloudFront logs, Mysql logs, access logs, Nginx logs, Http headers, user agent field, java stack trace.

Note that every regex is submitted as a recommendation, of course logs can have different configurations and permutations, you can easily adjust the parsing rules below to your needs, more on named group regex here.

1. User Agent (Use an “Extract” rule in Coralogix):

https://regex101.com/r/pw0YeT/3

Sample Log

Mozilla/5.0 (iPad; U; CPU OS 3_2_1 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Mobile/7B405

Regular Expression

(?P<mozillaVersion>Mozilla/[0-9.]+) ((?P<sysInfo>[^)]+))(?: (?P<platform>[^ ]+))?(?: ((?P<platformInfo>[^)]+)))?(?: (?P<extentions>[^n]+))?

Results

{ 
  "extentions" : "Mobile/7B405" ,
  "platformInfo" : "KHTML, like Gecko" ,
  "sysInfo" : "iPad; U; CPU OS 3_2_1 like Mac OS X; en-us" ,
  "mozillaVersion" : "Mozilla/5.0" ,
  "platform" : "AppleWebKit/531.21.10"
}

2. Cloud-Front (Use a “Parse” rule in Coralogix):

https://regex101.com/r/q2DmKi/4

Sample Log

2014-05-23 01:13:11 FRA2 182 192.0.2.10 GET d111111abcdef8.cloudfront.net /view/my/file.html 200 www.displaymyfiles.com Mozilla/4.0%20(compatible;%20MSIE%205.0b1;%20Mac_PowerPC) - zip=98101 RefreshHit MRVMF7KydIvxMWfJIglgwHQwZsbG2IhRJ07sn9AkKUFSHS9EXAMPLE== d111111abcdef8.cloudfront.net http - 0.001 - - - RefreshHit HTTP/1.1 Processed 1

Regular Expression

(?P<date_time>[0-9]{4}-[0-9]{2}-[0-9]{2}s*[0-9]{2}:[0-9]{2}:[0-9]{2}) (?P<x_edge_location>[^ ]+) (?P<sc_bytes>[0-9]+) (?P<c_ip>[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}) (?P<cs_method>[^ ]+) (?P<cs_host>[^ ]+) (?P<cs_uri_stem>[^ ]+) (?P<sc_status>[0-9]+) (?P<cs_referer>[^ ]+) (?P<cs_user_agent>[^ ]+) (?P<cs_uri_query>[^ ]+) (?P<cs_cookie>[^ ]+) (?P<x_edge_result_type>[^ ]+) (?P<x_edge_request_id>[^ ]+) (?P<x_host_header>[^ ]+) (?P<cs_protocol>[^ ]+) (?P<cs_bytes>[^ ]+) (?P<time_taken>[^ ]+) (?P<x_forwarded_for>[^ ]+) (?P<ssl_protocol>[^ ]+) (?P<ssl_cipher>[^ ]+) (?P<x_edge_response_result_type>[^ ]+) (?P<cs_protocol_version>[^ ]+) (?P<fle_status>[^ ]+) (?P<fle_encrypted_fields>[^n]+)

Results

{ 
  "x_edge_location" : "FRA2" , 
  "cs_method" : "GET" , 
  "x_edge_result_type" : "RefreshHit" , 
  "ssl_cipher" : "-" ,
  "cs_uri_stem" : "/view/my/file.html" , 
  "cs_uri_query" : "-" ,
  "x_edge_request_id" : "MRVMF7KydIvxMWfJIglgwHQwZsbG2IhRJ07sn9AkKUFSHS9EXAMPLE==" , 
  "sc_status" : "200" , 
  "date_time" : "2014-05-23 01:13:11" ,
  "sc_bytes" : "182" , 
  "cs_protocol_version" : "HTTP/1.1" ,
  "cs_protocol" : "http" , 
  "cs_cookie" : "zip=98101" , 
  "ssl_protocol" : "-" ,
  "fle_status" : "Processed" ,
  "cs_user_agent" : "Mozilla/4.0%20(compatible;%20MSIE%205.0b1;%20Mac_PowerPC)" ,
  "cs_host" : "d111111abcdef8.cloudfront.net" ,
  "cs_bytes" : "-" ,
  "x_edge_response_result_type" : "RefreshHit" ,
  "fle_encrypted_fields" : "1" ,
  "c_ip" : "192.0.2.10" ,
  "time_taken" : "0.001" ,
  "x_forwarded_for" : "-" ,
  "x_host_header" : "d111111abcdef8.cloudfront.net" ,
  "cs_referer" : "www.displaymyfiles.com"
 }

 

3. ELB (Elastic Load Balancer) – (Use a “Parse” rule in Coralogix):

https://regex101.com/r/T52klJ/1

Sample Log

2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 0.000086 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/ HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2

Regular Expression

(?P<timestamp>[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9A-Z]+) (?P<elbName>[0-9a-zA-Z-]+) (?P<clientPort>[0-9.:]+) (?P<backendPort>[0-9.:]+) (?P<request_processing_time>[.0-9-]+) (?P<response_processing_time>[.0-9]+) (?P<elb_status_code>[.0-9-]+) (?P<backend_status_code>[0-9-]+) (?P<received_bytes>[0-9-]+) (?P<sent_bytes>[0-9-]+) (?P<request>[0-9-]+) "(?P<user_agent>[^"]+)" "(?P<ssl_cipher>[^"]+)" (?P<ssl_protocol>[- A-Z0-9a-z.]+)

Results

{ 
  "received_bytes" : "200" , 
  "request" : "57" , 
  "elb_status_code" : "0.001337" , 
  "ssl_cipher" : "curl/7.38.0" , 
  "elbName" : "my-loadbalancer" ,
  "request_processing_time" : "0.000086" , 
  "sent_bytes" : "0" , 
  "response_processing_time" : "0.001048" , 
  "backendPort" : "10.0.0.1:80" , 
  "backend_status_code" : "200" , 
  "clientPort" : "192.168.131.39:2817" , 
  "ssl_protocol" : "DHE-RSA-AES128-SHA TLSv1.2" , 
  "user_agent" : "GET https://www.example.com:443/ HTTP/1.1" , 
  "timestamp" : "2015-05-13T23:39:43.945958Z"
}

4. MongoDB (Use a “Parse” rule in Coralogix):

https://regex101.com/r/pBM9DO/1

Sample Log

2014-11-03T18:28:32.450-0500 I NETWORK [initandlisten] waiting for connections on port 27017

Regular Expression

(?P<timestamp>[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}-[0-9]{4}) (?P<severity>[A-Z]) [A-Z]+ *[[a-zA-Z0-9]+] (?P<message>[^n]+)

Results

{ 
  "severity" : "I" , 
  "message" : "waiting for connections on port 27017" ,
  "timestamp" : "2014-11-03T18:28:32.450-0500" 
}

5. NSCA access logs (Use a “Parse” rule in Coralogix):

https://regex101.com/r/Iuos8u/1/

Sample Log

172.21.13.45 - MicrosoftJohnDoe [07/Apr/2004:17:39:04 -0800] "GET /scripts/iisadmin/ism.dll?http/serv HTTP/1.0" 200 3401

Regular Expression

(?P<clientIP>[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})s*(?P<userIdentidier>[^ ]+) (?P<userID>[^ ]+) [(?P<timestamp>[^]]+)] "(?P<clientRequest>[^"]+)" (?P<statusCode>[0-9-]+) (?P<numBytes>[0-9-]+)

Results

{ 
   "numBytes" : "3401" ,
   "userIdentidier" : "-" ,
   "clientIP" : "172.21.13.45" ,
   "userID" : "MicrosoftJohnDoe" ,
   "statusCode" : "200" , 
   "timestamp" : "07/Apr/2004:17:39:04 -0800" , 
   "clientRequest" : "GET /scripts/iisadmin/ism.dll?http/serv HTTP/1.0"
}

6. Java Stacktrace (Use an “Extract” rule in Coralogix):

https://regex101.com/r/ZAAuBW/2

Sample Log

Exception in thread "main" java.lang.NullPointerException 
at com.example.myproject.Book.getTitle(Book.java:16) 
at com.example.myproject.Author.getBookTitles(Author.java:25)
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

Regular Expression

Exception(?: in thread) "(?P<threadName>[^"]+)" (?P<changethenamelater>.*)s+(?P<stackeholder>(.|n)*)

Results

{ 
  "changethenamelater" : "java.lang.NullPointerException " ,
  "stackeholder" : "at com.example.myproject.Book.getTitle(Book.java:16) 
                    at com.example.myproject.Author.getBookTitles(Author.java:25) 
                    at com.example.myproject.Bootstrap.main(Bootstrap.java:14)" ,
  "threadName" : "main" 
}

7. Basic HTTP Headers (Use a “Extract” rule in Coralogix):

https://regex101.com/r/JRYot3/1

Sample Log

GET /tutorials/other/top-20-mysql-best-practices/ HTTP/1.1

Regular Expression

(?P<method>[A-Z]+) (?P<path>[^ ]+) (?P<protocol>[A-Z0-9./]+)

Results

{ 
  "path" : "/tutorials/other/top-20-mysql-best-practices/" , 
  "protocol" : "HTTP/1.1" , 
  "method" : "GET"
}

8. Nginx (Use a “Parse” rule in Coralogix):

https://regex101.com/r/yHA8Yh/1

Sample Loghttps://regex101.com/r/yHA8Yh/1

127.0.0.1 - dbmanager [20/Nov/2017:18:52:17 +0000] "GET / HTTP/1.1" 401 188 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0"

Regular Expression

(?P<remoteAdd>[0-9.]+) - (?P<remoteUser>[a-zA-z]+) [(?P<timestamp>[^]]+)] "(?P<request>[^"]+)" (?P<status>[0-9]+) (?P<bodyBytesSent>[0-9]+) "(?P<httpReferer>[^"]+)" "(?P<httpUserAgent>[^"]+)"

Results

{ 
  "remoteUser" : "dbmanager" , 
  "request" : "GET / HTTP/1.1" ,
  "bodyBytesSent" : "188" , 
  "remoteAdd" : "127.0.0.1" , 
  "httpUserAgent" : "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:47.0) Gecko/20100101
   Firefox/47.0" ,  
  "httpReferer" : "-" , 
  "timestamp" : "20/Nov/2017:18:52:17 +0000" ,
  "status" : "401"
}

9. MySQL (Use a “Parse” rule in Coralogix):

https://regex101.com/r/NjtRLZ/4

Sample Log

2018-03-31T15:38:44.521650Z 2356 Query SELECT c FROM sbtest1 WHERE id=164802

Regular Expression

(?P<timestamp>[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{6}Z) ? ? ? ? ? ?(?P<connID>[0-9]+) (?P<name>[a-zA-Z]+) (?P<sqltext>[^n]+)

Results

{
“sqltext” : “SELECT c FROM sbtest1 WHERE id=164802” ,
“name” : “Query” ,
“connID” : “2356” ,
“timestamp” : “2018-03-31T15:38:44.521650Z”
}

10. ALB (Use a “Parse” rule in Coralogix):

https://regex101.com/r/NjtRLZ/4

Sample Log

2018-03-31T15:38:44.521650Z 2356 Query SELECT c FROM sbtest1 WHERE id=164802 http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 502 - 34 366 "GET https://www.example.com:80/ HTTP/1.1" "curl/7.46.0" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 "Root=1-58337364-23a8c76965a2ef7629b185e3" "-" "-" 0 2018-11-30T22:22:48.364000Z "forward" "-" "LambdaInvalidResponse"

Regular Expression

(?P<type>[a-z0-9]{2,5}) (?P<timestamp>[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{6}Z) (?P<elb>[^n]+) ? ?(?P<clientPort>[0-9.:-]+) (?P<targetPort>[0-9.:-]+) (?P<requestProcessTime>[0-9.:]+) (?P<targetProcessTime>[0-9.:-]+) (?P<responseProcessingTime>[0-9.:-]+) (?P<elbStatusCode>[0-9-]+) (?P<targetStatus>[0-9-]+) (?P<recievedBytes>[0-9-]+) (?P<sentBytes>[0-9-]+) ? ?"(?P<request>[^"]+)" "(?P<userAgent>[^"]+)" (?P<sslCipher>[^ ]+) (?P<sslProtocol>[^n]+) ? ?(?P<targetGroupArn>[^n]+) ? ?"(?P<traceID>[^"]+)" "(?P<domainName>[^"]+)" "(?P<chosenCertArn>[^"]+)" ? ?(?P<matchedRulePriority>[^ ]+) (?P<requestCreationTime>[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{6}Z) "(?P<actionsExecuted>[^"]+)" "(?P<redirectURL>[^"]+)" "(?P<errorReason>[^"]+)"

Results

{

“traceID” : “Root=1-58337364-23a8c76965a2ef7629b185e3” ,
“request” : “GET https://www.example.com:80/ HTTP/1.1” ,
“requestCreationTime” : “2018-11-30T22:22:48.364000Z” ,
“redirectURL” : “-” , “targetGroupArn” : ” ” ,
“type” : “http” , “targetPort” : “-” ,
“responseProcessingTime” : “0.000” ,
“targetProcessTime” : “0.001” ,
“chosenCertArn” : “-” ,
“errorReason” : “LambdaInvalidResponse” ,
“matchedRulePriority” : “0” ,
“actionsExecuted” : “forward” ,
“clientPort” : “7” ,
“elb” : “app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:281” ,
“targetStatus” : “-” ,
“recievedBytes” : “34” ,
“timestamp” : “2018-11-30T22:23:00.186641Z” ,
“sslCipher” : “-” ,
“userAgent” : “curl/7.46.0” ,
“requestProcessTime” : “0.000” ,
“domainName” : “-” ,
“elbStatusCode” : “502” ,
“sslProtocol” : “- arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067” ,          “sentBytes” : “366”
}

11. IIS (Use a “Parse” rule in Coralogix):

https://regex101.com/r/ytJdyE/2

Sample Log

192.168.114.201, -, 03/20/05, 7:55:20, W3SVC2, SERVER, 172.21.13.45, 4502, 163, 3223, 200, 0, GET, /DeptLogo.gif, -,

Regular Expression

(?P<clientIP>[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}), (?P<userName>[^,]+), (?P<timestamp>[[0-9]{2}/[0-9]{2}/[0-9]{2}, [0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}), (?P<serviceInstance>[^,]+), (?P<serverName>[^,]+), (?P<serverIP>[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}), (?P<timeTaken>[^,]+), (?P<clientBytesSent>[^,]+), (?P<serverBytesSent>[^,]+), (?P<serviceStatusCode>[^,]+), (?P<windowsStatusCode>[^,]+), (?P<requestType>[^,]+), (?P<targetOfOperation>[^,]+), (?P<parameters>[^,]+),

Results

{
“requestType” : “GET” ,
“windowsStatusCode” : “0” ,
“serverName” : “SERVER” ,
“userName” : “-” ,
“timeTaken” : “4502” ,
“serverBytesSent” : “3223” ,
“clientIP” : “192.168.114.201” ,
“serverIP” : “172.21.13.45” ,
“serviceInstance” : “W3SVC2” ,
“parameters” : “-” ,
“targetOfOperation” : “/DeptLogo.gif” ,
“serviceStatusCode” : “200” ,
“clientBytesSent” : “163” ,
“timestamp” : “03/20/05, 7:55:20”
}

12. Apache (Use a “Parse” rule in Coralogix):

https://regex101.com/r/2pwM6J/1

Sample Log

127.0.0.1 – frank [10/Oct/2000:13:55:36 -0700] “GET /apache_pb.gif HTTP/1.0” 200 2326

Regular Expression

(?P<clientIP>[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}) (?P<identd>[^ ]+) (?P<userid>[^ ]+) [(?P<timesptamp>[^]]+)] "(?P<request>[^"]+)" (?P<statusCode>[^ ]+) (?P<objectSize>[^ ]+)

Results

{
“request” : “GET /apache_pb.gif HTTP/1.0” ,
“timesptamp” : “10/Oct/2000:13:55:36 -0700” ,
“objectSize” : “2326” ,
“clientIP” : “127.0.0.1” ,
“identd” : “-” ,
“userid” : “frank” ,
“statusCode” : “200”
}