Tag: data platform

Coralogix’s Cross-Vendor Compatibility To Keep Your Workflow Smooth

Posted on by eugene evdokimov

Coralogix supports logs, metrics, traces and security data, but some organizations need a multi-vendor strategy to achieve their observability goals, whether it’s developer adoption, or vendor lock-in is preventing them from migrating all of their data.

Coralogix offers a set of features that allow customers to bring all of their data into a single flow—across SaaS and hosted solutions. 

Custom actions: Bring extensibility to your data

Custom actions in Coralogix let customers define bespoke functionality within the platform, such as redirect their browser to another SaaS vendor, a hosted solution within their VPN or another application entirely. Custom actions enable extensible observability, which is the key to a cross-vendor strategy.

How does it work?

Custom actions allow Coralogix customers to populate URL templates with values from the data. For example, a custom action attached to a log can template a URL with values from within the log document, or any related metadata. 

Joining Coralogix logs to DataDog metrics

A customer may host their logs in Coralogix, but have their infrastructure metrics in DataDog. Normally, this would lead to a fragmented and confusing debugging process, however Coralogix makes this hybrid process easy. 

Using custom actions, Coralogix customers can easily connect multiple solutions together, with one click integrations. This encourages a true multi-vendor strategy and makes it as easy as possible to get the greatest possible benefit from every tool for which your organization is paying. 

Connecting Coralogix metrics to New Relic infrastructure monitoring

Metrics can also be connected using Coralogix’s custom actions. Customers can define simple integrations from their metrics views, like DataMap, and instantly jump to any other view they want. For example, to the Infrastructure and Kubernetes monitoring in New Relic.

Coralogix. Ready for anything, open to everyone.

At Coralogix, we don’t believe in locking our customers in, whether that’s with proprietary agents, custom data formats or obscure pricing. Instead, we favor open source integrations, use open source data formats and have the most effective pricing model on the market. 

That also goes for our integrations. Coralogix boasts a comprehensive featureset, processing logs, metrics, traces and security data. You can also convert that into tailored infrastructure monitoring, custom dashboards, data transformation pipelines and much more.

Our goal is to provide a seamless experience for all of our customers, giving them the tools they need to solve any problem along the way. Coralogix doesn’t just integrate cleanly with open source tooling. Our custom actions integrate with competitors’ platforms to keep your observability workflow smooth.  

Creating a Free Data Lake with Coralogix

Posted on by eugene evdokimov

Like many cool tools out there, this project started from a request made by a customer of ours.

Having recently migrated to our service, this customer had ~30TB of historical logging data.
This is a considerable amount of operational data to leave behind when moving from one SaaS platform to another. Unfortunately, most observability solutions are built around the working assumption that data flows are future-facing.

To put it in layman’s terms, most services won’t accept an event message older than 24 hours. So, we have this customer coming to us and asking how we can migrate this data over, but those events were over a year old! So, of course, we got to thinking…

Data Source Requirements

This would be a good time to describe the data sources and requirements that we received from the customer.

We were dealing with:

  • ~30 TB of data
  • Mostly plain text logs
  • Various sizes of gzip files from 1GB to 200KB
  • A mix of spaces and tabs
  • No standardized time structure
  • Most of the text represents a certain key/value structure

So, we brewed a fresh pot of coffee and rolled out the whiteboards…

Sending the Data to Coralogix

First, we created a small bit of code to introduce the log lines into Coralogix. The code should work in parallel and be as frugal as possible.

Once the data is coming into Coralogix, the formatting and structuring of the data can be done by our rules engine. All we needed is to extract the timestamp, make it UNIX compatible, and we are good to go.

We chose to do this by implementing a Lambda function with a SAM receipt. The Lambda will trigger for each S3 PUT event so we can have a static system costing us nothing on idle and always ready to handle any size of data dump we throw its way.

You can check out the code on our GitHub.

Standardizing Timestamps

Now that we have the data streaming in, we need to make sure it keeps its original timestamp.
Don’t forget it basically has two timestamps now:

  • Time of original message
  • Time of entry to Coralogix

In this part, we make the past timestamp the field by which we will want to search our data.

Like every good magic trick, the secret is in the moment of the swap, and for this solution this is it.

Since we have the original time stamp, we can configure it to be of a Date type. All we need to do in Coralogix is make sure the field name has the string timestamp in its name
(i.e. coralogix_custom_timestamp).

Some parts of Coralogix are based on the community versions of Elastic stack, so we also place some of the advanced configurations at the user’s disposal (i.e. creation of index templates or Kibana configurations).

Creating Index Patterns

At this point, we need to create a template for new indexes to use our custom timestamp field.

While the Elastic engine will detect new fields and classify them accordingly by default, we can override this as part of the advanced capabilities of Coralogix.

Once this part is done, we will have the ability to search the “Past” in a native way. We will be able to set an absolute time in the past.

To create the templates, click on the Kibana logo on the top right of the Coralogix UI ->  select Management -> Index Patterns. This is essentially where we can control the template which creates the data structures of Coralogix. 

First, we should remove the current pattern template (i.e. *:111111_newlogs*).

Note – This step will only take effect on the creation of the new index (00:00:00 UTC).

Clicking on Create index format, one of the first parameters we are asked to provide is the field which will indicate the absolute time for new indices. In this case, “Time filter field name”.

If using the example field name suggested earlier, the field selected should be “coralogix_custom_timestamp”.

Sending Data to S3

Now that we have a team with flowing historical data and a time axis aware of the original time, all we have left is to point the Coralogix account to an S3 bucket to grant us endless retention. Essentially, the data goes through Coralogix but does not stay there. 

For this step, we will use our TCO optimizer feature to configure a new policy for the application name we set on our Lambda. This policy will send all of our data to our S3 bucket.

Now the magic is ready!

Wrapping Up

Once a log gzip file is placed in S3, it will trigger an event for our Lambda to do some pre-parsing for it, and send it to Coralogix.

As data flows through Coralogix, it will be formatted by the rules we set for that application.

The data will then be structured and sent to S3 in a structured format. Once the data is sent to S3, it is no longer stored in the Coralogix platform in order to save on storage costs. You can still use Athena or any search engine to query the data with low latency. Behold! Your very own data lake was created with the help of Coralogix. If you have any questions about this or are interested in implementing something similar with us, don’t hesitate to reach out! 

Unlocking Hidden Business Observability with Holistic Data Collection

Posted on by eugene evdokimov

Why do organizations invest in data observability?

Because it adds value. Sometimes we forget this when we’re building our observability solutions. We get so excited about what we’re tracking that we can lose sight of why we’re tracking it.

Technical metrics reveal how systems react to change. What they don’t give is a picture of how change impacts the broader business goals. The importance of qualitative data in business observability is often overlooked.

Data-driven insights which only include technical or quantitative modeling miss the big picture. Pulling data from holistic sources unlocks the full power of business observability.

Including holistic data collectors in your observability stack grants visibility not just into what’s happening, but why it happened, and the impact it has on your business outside of your systems.

What are Holistic Data Collectors?

Holistic data collectors draw from unusual or uncommon sources. They log information that wouldn’t usually be tracked, enabling businesses to get a holistic view of their systems. A holistic view means observability of all interconnected components.

A data strategy that includes the collection of holistic data empowers effective business observability. By logging as hidden pockets of data much clearer insight can be generated, and data-backed strategic decisions become much better informed.

The list of data sources it is possible to include is potentially limitless. With the proper application of collectors, any communication platform or user service can become a source of data and insight. Holistic data collectors exist for code repositories such as GitHub, collaboration tools like Teams or Slack, and even marketing data aggregators such as Tableau.

When and How to Use Holistic Data Collection

With some creative thinking and technical know-how, almost anything can be a source of holistic data. Common business tools, software, and platforms can be mined for useful data and analysis.

Below are some examples that exemplify and illustrate the usefulness of this approach to data-driven insight.

Microsoft Teams

Microsoft Teams has become a vital communication channel for the modern enterprise. As one of the most popular internal communication platforms on the market, Teams can be an invaluable source of holistic data from within your workforce.

Integrating webhooks into Teams enables you to monitor and track specific activity. Webhooks are a simple HTTP callback, usually in JSON format. They’re one of the simplest ways to connect your systems to an externally hosted channel such as Teams.

Pushing holistic, qualitative Teams data to a third party platform using webhooks enables correlation of non-numeric insight with finance and marketing figures and system metrics. 

As Teams is most often used internally, this is an invaluable asset for understanding how changes to your organization are reflected in the morale of your staff. Many developers use Teams to discuss issues and outages. Having visibility of your IT team’s responses identifies which tasks take the most of their time and what knowledge blindspots exist in their collective technical skillset.

PagerDuty

PagerDuty is transforming the way IT teams deal with incidents. Integrating holistic data collectors greatly expands the level of insight gained from this powerful tool.

High levels of criteria specificity on alerts and push notifications to enable effective prioritizing of support. IT teams can manage large sets of alerts without risking an overwhelming amount of notifications.

As with Teams, webhooks are one of the most common and simplest methods of collecting holistic data from PagerDuty. By collecting enriched event data around incidents, outages, and how your IT teams respond can be analyzed in the context of the wider business organization.

GitHub

Scraping GitHub provides great insight into the performance of your dev teams. What’s not as widely known is the business insight that can be gained by correlating GitHub commits on repositories.

Commits are changes to code in GitHub. Each commit comes with a comment message and log. Keeping GitHub commits under the eye of your monitoring stack reveals a fascinating hidden pocket of data that could change the way your teams approach outages and troubleshooting.

Outages occur for many reasons. Some require more work and code changes than others. Bad or lazy coding creates a lot of outages. Tracking and logging GitHub commits will reveal both the kinds of outages and the specific chunks of code that take up the most time for your engineers.

GitOps

Monitoring GitOps declarations and version edits pinpoint not only where weaknesses exist at an architectural level, but when they were implemented and whether the problem is isolated or part of a wider trend.  

Tableau

Tableau is an invaluable source of marketing data. Integrating Tableau with your log analytics platform opens up integral insight.

Digital marketing is an essential business aspect of modern enterprises. An effective digital marketing presence is key to success, and Tableau is the go-to tool for many organizations.

Tableau is useful for market strategy. It’s when Tableau data is included as part of a wider, holistic picture that organizational intelligence and insight can be gained from it. Scraping Tableau for data such as page bounce rates and read time allows you to see how technical measurements correlate with your marketing metrics.

Silo-Free Reporting

Say you’ve experienced a sudden dip in online sales despite an advertising drive. Your first reaction could be to blame the marketing material.

By including Tableau data in your analytics and reporting you can see that the advertising campaign was successful. Your website had a spike in visitors. This spike in traffic led to lag and a poor customer experience, visible due to an equally large spike in bounce rate. 

Scraping Tableau as a source of holistic data reveals that the sales dip was down to issues with your systems. Your strategy can then be to improve your systems so they can keep up with your successful marketing and large digital presence.

Jira

Integrating your analytics and monitoring platform with Jira can yield powerful results, both in alerting capabilities and collecting insight-generating data.

Using webhooks, your integrated platform can create Jira tickets based on predefined criteria. These criteria can be defined based on data from other sources in your ecosystem, as the issue is being raised and pushed from a third party platform.

Automating this process enables your IT team to deploy themselves both efficiently and with speed. It allows them to concentrate on resolving issues without getting bogged down in logging and pushing the alerts manually.

By having tickets raised by an ML-powered platform with observability over your whole infrastructure, your engineers won’t be blindsided by errors occurring in areas they may not have predicted.  

Use Case: Measuring the Business Impact of Third Party Outages with Holistic Collectors

Third party outages are one of the most common reasons for lost productivity and revenue. Many enterprises rely on third party software, systems, or services to deliver their own. Reliance on one or more third parties is the norm for many sectors and industries.

A third party will inevitably experience an outage at some point.  There’s no way to avoid this. Even the most reliable provider can fall victim to unforeseen circumstances such as power cuts or emergency server maintenance from time to time.

Third party outages can have huge ramifications. These services are often business-critical, and the financial costs of even brief outages can reach far past the hundred thousand dollar mark. 

Lost revenue is one way to measure the impact of such an outage. While financial data helps understand the initial impact of unforeseen downtime, alone it doesn’t provide full visibility of long-term consequences or the reaction of the business and consumers.

Collecting holistic data alongside the usual logging metrics helps to fill in the blanks. It allows a business to answer questions like:

  • Has this affected our public reputation?
  • Are users speaking positively about our response?
  • Did we respond faster or slower to this outage than usual?
  • Was our response effective at returning us to BAU as soon as possible?
  • Is this affecting the morale of our staff?
  • How much work is this making for my IT team?
  • Has web traffic taken a hit?

This leaves any business or organization in a much better position to control and mitigate the long-term impact of a third party outage. 

A study by Deloitte found that direct losses due to third party mismanagement have been found to directly cost businesses as much as $48m. This is before indirect losses are factored in. An outage could have minor immediate financial ramifications but damage long-term prospects through reputational damage. It would be almost impossible to gain the insight to prevent this using financial or systems metrics alone.

A Complete Holistic Data Solution

The Coralogix observability platform is a monitoring solution that enables holistic data collection from sources such as Teams, GitHub, PagerDuty, Slack, Tableau, and many others.

Collecting and logging data from multiple sources, both traditional and unorthodox, can be difficult to manage. Business intelligence and organizational insight are difficult to gain if information is stored and reported from dozens of sources in as many formats.

Coralogix’s observability platform provides a holistic system and organized view in a single pane. Using machine learning, our platform creates a comprehensive picture of your technical estate that comes with actionable business context. For visibility and insight that extends beyond the purely technical, the Coralogix platform is the solution your business needs.