What are AWS Log Insights and How You Can Use Them

Within this blog post, we’re going to take a look at AWS Log Insights and cover some of the topics that you will find useful around what it is, how to use it, and how it can link in with our various solutions. 

Why do you need log insights in AWS?

When you have a couple of servers, your logging capability doesn’t need to be particularly complex. However, AWS monitoring makes it very, very easy to scale up your servers. Soon, you’ll have lots of different instances, and they’ll all be performing different tasks. Moreover, these instances may appear and disappear as they autoscale.

This increased complexity demands a more sophisticated observability system, and a fundamental part of that is your logging capability. Insights from logging can provide a great deal of context around the behavior of your system and give you that all-important revelation when you’re dealing with an outage. So how do you implement your log insights in AWS, and why are AWS log insights becoming the de facto standard?

We’ll begin with Cloudwatch

Cloudwatch is the observability system in AWS. Imagine if you are running an Nginx or Apache system on an EC2 instance. You’d use the CloudWatch Agent to collect the additional data that is not out of the box so this data is stored within CloudWatch, then you’d use CloudWatch Log Insights to analyze this data to produce meaningful information about what is happening. The different pieces of the puzzle are all interlinked and designed to make your life easy. CloudWatch Log Insights is a paid service from Amazon and is designed to be an interactive and fully integrated pay-as-you-go logging analytics service. 

What this means is that your ability to utilize CloudWatch Log Insights is heavily dependent on the data that you can initially get flowing through into CloudWatch. So take some time to get these foundations right in the first instance. 

To visualize the architecture of the AWS CloudWatch Log Insights, the below diagram outlines how the data is feeding on multiple source systems, into CloudWatch, from there then the data can be queried to produce the insights you need;

AWS CloudWatch is capable of automatically capturing information from supported AWS services such as Amazon VPC Flow Logs, Route 53 Logs, Lambda Logs, CloudTrail Logs, and other logs in JSON format. Off the back of this captured information, AWS CloudWatch Insights is able to query this information due to the data now being stored in a structured format including information about the @message, @timestamp, @ingestionTime, @logStream, and @log. 

Can Coralogix help you with this data?

Coralogix integrates directly with Cloudwatch, so you can build a machine learning-powered observability toolset, off the back of this data. This addition to Cloudwatch means you can get the native benefit of AWS log collection, combined with the analytical power of the Coralogix feature set.

CloudWatch Log Insights Query Language

As we just touched on, CloudWatch Log Insights comes with a querying language to allow you to interrogate your data in CloudWatch. If you are familiar with one of the many flavors of SQL database query languages, you’ll soon pick up the nuances that come with this one. 

An often-overlooked element of logging solutions is the usability of their query language. Cloudwatch Log insights overcome this by creating an IDE-style experience, offering autocomplete and syntax checking on the fly. This makes it far more enjoyable to interrogate your logs. For those of you who have a background in a range of languages, this will no doubt be of huge benefit to avoid having to remember all the nuances of yet another language.  

To give you a flavor of the type of things you can do with the CloudWatch Log Insights Query Language, let’s look at a few of the commands and what they do;

• display: Specifies the fields to return in the query results
• fields: Retrieves the specific fields from the log event
• filter: Filters the results based on the conditions you set
• stats: Allows you to utilize aggregate functions
• sort: Allows you to order your results
• limit: Enables you to restrict the number of results that are returned
• parse: Function that enables you to extract specific data by utilizing regular expressions

Within the query language you also have the ability to utilize various operations and functions including;

• Comparison operators: Such as =, !=, <, <=, >, >=
• Boolean operators: Such as and, or and not
• Arithmetic operations
• Numeric operations: Such as getting the absolute value, rounding, square root, and more
• String functions: Such as isempty, isblank, concat, and many more
• Datetime functions
• IP Address Functions: To determine the IP address from where the log event was triggered

One of the major challenges with CloudWatch Log Insights is that it is extremely flexible in nature. The challenges come with setting everything up from the outset. The query language examples that you can find online are great for basic things, but if you venture beyond that, you are pretty much on your own. AWS Log Insights can be difficult to navigate at times, which is why Coralogix supports industry-standard querying languages such as Apache Lucene. 

Summary

Hopefully, this has been a valuable overview of CloudWatch Log Insights to help you understand what it is and how it can benefit you and your organization to bring your data to life. As a final point around how you can utilize Coralogix for your organization, the platform comes with full control over your pricing. Simply pay for what you have with our tiered storage along with the ability to utilize querying technology against compressed and unindexed logs. Get in touch to request a demo.