How IaC helps integrate Coralogix with Terraform

Infrastructure as Code is an increasingly popular DevOps paradigm. IaC has the ability to abstract away the details of server provisioning. This tutorial will look at how Coralogix can be used with the popular IaC tool Terraform.

Terraform

Terraform, a tool we’ve previously talked about, is Hashicorp’s answer to the problem of server provisioning.  It uses the powerful paradigm of Infrastructure as Code (IaC). IaC abstracts and simplifies the traditional process of setting up and configuring servers by representing server configurations as code files.

This brings a range of benefits to DevOps teams such as automating deployment processes, providing effective infrastructure documentation, and enabling infrastructure validation.

Terraform itself is a binary that makes API calls to providers. These are services such as Coralogix or AWS which Terraform tells to perform particular tasks. Users can interact with the providers using the Terraform CLI or by setting up configuration files.

Terraform Configuration Language

Terraform represents the various objects of your infrastructure as resources. These are stored in configuration documents. The syntax of a typical configuration document looks something like this:

resource "aws_vpc" "main" {
  cidr_block = var.base_cidr_block
}

Terraform language components come in three types: blocks, arguments, and expressions.  

Blocks

Blocks are containers for other content.

<BLOCK TYPE> "<BLOCK LABEL>" "<BLOCK LABEL>" {
  # Block body
  <IDENTIFIER> = <EXPRESSION> # Argument
}

Blocks are comprised of three elements. The Block Type tells you that it is a Terraform resource. Block labels function as tags and a block can have multiple labels.  The body of a block is where the content is stored. This content could be arguments or expressions.

Arguments and expressions

Arguments assign a value to a name while expressions are statements that combine values. In the first example cidr_block = var.base_cidr_block is an argument.

Terraform’s configuration language is declarative.  As Yevgeniy Brikman explains, this shows its advantage when making configuration changes. For example, if you wanted to deploy 10 EC2 instances you might use the following configuration file:

resource "aws_instance" "example" {
  count = 10
  ami = "ami-40d28157"
  instance_type = "t2.micro"
}

You can change the number simply by editing the count argument. If you wanted 15 EC2 instances instead of ten you can write count = 15 without worrying about the configuration change history.

Using Terraform

Terraform has a range of useful applications.  For one, it can simplify the setup of Heroku applications. Heroku is popular due to its ability to scale apps using dynos, but building anything complex quickly requires lots of add-ons.

Terraform, by using IaC, can make setting up these add-ons much simpler. Heroku add-ons can be specified in a Terraform configuration document. Terraform even allows you to do fancy stuff like using Cloudflare as a CDN for your application.

Another use is 2-tier applications that involve a pool of web servers using a database.  For these to run successfully the connection between servers and database must be seamless.  Additionally, both tiers must be up and running to execute functionality. You don’t want any of your servers trying to hit a database that isn’t there.

With IaC, Terraform can handle the infrastructure, ensuring the necessary dependencies are in place and that the database is up before servers are provisioned. Plus, this can all be done with a few configuration documents. 

Coralogix

Many Terraform applications produce lots of logging data. As a case in point, Heroku logs are notorious for the amount of data they generate.  To really reap the benefits of IaC in your application development, you need good observability.

This is where Coralogix (which has pre-built Heroku visualizations) comes in. It uses machine learning to automatically extract patterns and trends from data.  

Using Coralogix with Terraform

As with other systems that it integrates with, you can use Terraform to interact with Coralogix, first by configuring it with the Coralogix provider, and second, by setting rules and alerts through the Terraform CLI.

Coralogix Provider

The Coralogix provider enables you to define rules and alerts through Terraform’s IaC paradigm.  If you have Terraform at or later than 0.13, the code for the provider is:

terraform {
  required_providers {
    coralogix = {
      source  = "coralogix/coralogix"
      version = "~> 1.0"
  }
 }
}

If you have Terraform 0.12 or earlier, the following code should be used:

# Configure the Coralogix Provider
provider "coralogix" {
    api_key = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

The value of the API key is stored in an environment variable called API_KEY. If you’re an admin user, you can generate an API key from the Coralogix dashboard by going to Settings -> Account and clicking on API Access. This will let you create an Alerts & Rules API key.Since the API key is a sensitive value you can use infrastructure as code management platform to store the value securely.

Along with the API key, there are two optional arguments. url contains the Coralogix API URL which is stored in the environment variable API_URL. timeout is an argument specifying when the Coralogix API will time out. This information is stored in the CORALOGIX_API_TIMEOUT environment variable.

Log Parsing Rules

It’s important for DevOps engineers to effectively manipulate logging data. In Coralogix, this is enabled through log parsing rules.  These are rules for processing, parsing, and restructuring log data. Rules come in various types, for example, parse rules allow you to create secondary logs based on data from primary logs.

Coralogix contains rules in Rules Groups. These are structures that contain sets of rules, along with a Rule Matcher which ensures only the desired logs are processed by queries.

Manipulating Rules Groups with Terraform

Terraform allows users to create, read, update, and delete Coralogix Rules Groups through its Coralogix Rules Groups resource.

In this example, we are creating a group called “My Group”.

# Create "My Group" Rules Group
resource "coralogix_rules_group" "rules_group" {
    name    = "My Group"
    enabled = true
}

In addition to the arguments included in the example, there are two optional arguments. The description argument allows you to add a description summarizing the Group’s purpose. The creator argument shows who created the rules group.

Manipulating Rules

Terraform lets you play not just with Rules Groups but with Rules themselves using this data source.

data "coralogix_rule" "rule" {
    rule_id        = "e1a31d75-36ab-11e8-af8f-02420a00070c"
    rules_group_id = "e10ef9d1-36ab-11e8-af8f-02420a00070c"
}

Rules can be created in the following way.

# Create "My Rule" Rule
resource "coralogix_rule" "example" {
    rules_group_id = "e10ef9d1-36ab-11e8-af8f-02420a00070c"
    name           = "My Rule"
    type           = "extract"
    description    = "My Rule created with Terraform"
    expression     = "(?:^|[\\s\"'.:\\-\\[\\]\\(\\)\\{\\}])(?P<severity>DEBUG|TRACE|INFO|WARN|WARNING|ERROR|FATAL|EXCEPTION|[I|i]nfo|[W|w]arn|[E|e]rror|[E|e]xception)(?:$|[\\s\"'.:\\-\\[\\]\\(\\)\\{\\}])"
    rule_matcher {
        field      = "applicationName"
        constraint = "prod"
    {
}

As with Rules Groups, rules have a name, description, and enabled flag.  They also have three other arguments.

Rules_group_id contains the id of the rules group that the rule belongs to. This allows users to know what Rules Group a rule is part of and re-assign rules to different rules groups.

The type specifies what type the rule is.  As explained at the beginning of this section, log parsing rules can come in different types. In this case, the rule type is “extract,” meaning that the rule is designed to extract information to a log and append additional fields to it.

The expression contains the rule itself in the form of a regular expression. In the above example, the rule is designed to search for logs containing words including DEBUG, TRACE, WARNING, and EXCEPTION.

Alerts

A key feature of Coralogix is the ability to create alerts. They enhance observability by alerting DevOps engineers whenever a parameter leaves its optimal state. Terraform lets users define Coralogix alerts with the Coralogix alert resource.

data "coralogix_alert" "alert" {
    alert_id        = "3dd35de0-0e10-11eb-9d0f-a1073519a608"
}

Here is how to create an alert.

# Create "My Alert" Alert
resource "coralogix_alert" "example" {
    name     = "My Alert"
    severity = "info"
    enabled  = true
    type     = "text"
    filter {
        text         = ""
        applications = []
        subsystems   = []
        severities   = []
}
    condition {
        condition_type = "more_than"
        threshold      = 100
        timeframe      = "30MIN"
}
    notifications {
        emails = [
            "[email protected]"
        ]
    }
}

Just like Rules and Rules Groups, each alert has a name argument and an enabled flag. Moreover, there are plenty of additional arguments to determine the properties of the alert.

There are two required arguments in addition to the name and enabled.

The type determines the alert type. This can be either “text” or “ratio.” Text alerts simply provide a message when a system parameter exceeds a certain threshold. For example, Coralogix provides dynamic alerts, which update the threshold using machine learning.

Ratio alerts are slightly more complex. They let you calculate a ratio between two log queries, something that can be useful in areas ranging from system health to marketing.

Severity specifies the alert’s urgency. It can take three values, which include the following –  “Info” means the alert simply provides information and the user is under no pressure to act on it. “Warning” is for when the alert provides a warning such as disk space is about to be used up. “Critical” would be for alerts that require immediate action, such as a system outage.

There are four block arguments; arguments whose values require Terraform Configuration Language blocks.

The filter defines what input the alert needs to respond to. This could be particular logs or application behavior. The block contains four optional fields. Text specifies the string query to be alerted on. 

Users can decide what applications and subsystems the alert should respond to with the applications and subsystems fields. The severities field enables users to list the log severity levels they want to be alerted on.

Condition is where users can define the threshold that triggers the alert. It has three required fields; condition_type works like a relational operator in Java or Python and threshold specifies the number of log occurrences that should trigger the alert. Timeframe determines how long after the event the alert can be triggered.

Schedule determines when the alert should be triggered while notifications control who gets notified about the alert. 

Wrapping Up

In this tutorial, we’ve seen how Coralogix can be used in tandem with the popular IaC tool Terraform.  Coralogix can be integrated with Terraform through the Terraform Coralogix provider and Terraform provides plenty of enabling features to use key aspects of Coralogix, like rules and alerts.

The power of Infrastructure as Code is that it allows you to configure DevOps infrastructure with the same ease that you write code. Being able to apply that to observability is a very powerful tool.