How Capgemini Solved Multi-Cloud Observability on Heroku/Salesforce

The modern enterprise has expanded its reach by using the power of cloud computing. However, with that power comes complexity in leveraging the multiple platforms needed to provide rich functionality. To achieve a seamless integration that involves multiple cloud infrastructures you need insightful and actionable data. You also need the right team to bring the clouds together in a seamless, effective, and efficient manner. Coralogix together with Capgemini provided the expertise and technology for a recent project that needed full multi-cloud observability across Heroku/Salesforce.

The Observability Challenge 

This project shows how with the right team and technology in place, even the most complex, multi-functional, cross-platform system can be made to mesh seamlessly. The customer, a large European automotive manufacturer with a global presence, required a rollout of their global CRM system, based on Heroku Platform-as-a-Service (PaaS). The platform needed to be dynamic scalability and would see significantly increased functionality; the goal, to weave together over 100 applications in a single deployment.

The system was mass-adopted with over 20,000 users as well as thousands of vendors in the supply chain. The key challenge of the project was a major undertaking: Existing sales data must dovetail with a new CRM (Customer Relationship Management) solution through a dedicated integration layer for all inbound and outbound data streams.

Several platforms were behind the customer’s ecosystem: 

• Salesforce Sales Cloud
• Salesforce Marketing Cloud
• MuleSoft
• Heroku
• Customer liaison systems
• Identity platforms

The Solution 

The team that successfully took on the challenge was made up of core specialists, Capgemini, Salesforce Heroku, and Coralogix. Working in collaboration, the team builds dedicated multi-cloud platforms for customers that meet exacting and complex needs.

The Role of Coralogix in the Customer Challenge 

To meet the requirements of this major and complex project, Coralogix met a list of criteria for integration across the multi-cloud platform. The core Coralogix functionality powering the observability project included:

Dynamic alerts 

Dynamic alerts offer a way to define the threshold for an alert trigger. Coralogix provides even finer granularity as thresholds can be time-based, changing over the course of a day. Ultimately, this reduces alert overload and creates greater focus and accuracy. 

Rules

The project utilizes Coralogix rules that are used to provide structure to unstructured logs. These unstructured logs are formatted to JSON, which allows for easier and controllable queries and visualizations. The resulting alerts can be used with query-based detection to proactively identify issues and shorten resolution time.

Contextual logs

Logs provide context for application lifecycle events. This offers a way to easily understand the impact of new features on the system.  For example, during the project, when a problem was identified an automatic tag was created. This is then used to cross-reference against when troubleshooting, helping to reduce resolution time. Notably, one of the latest Coralogix features is the ability to verify an LDAP configuration dynamically, based on the previous 24 hours logs; this helps in finding the threshold for alerts.

Machine Learning (ML)

ML capabilities empower Coralogix with the ability to decipher key information from data patterns. The Coralogix ML identifies statistically significant errors during a process flows sending out an alert if issues are detected. 

Coralogix Machine learning is used for troubleshooting issues in the project, and to provide data for root cause analysis. Two key methods are used in the project: 

• Loggregation: a process that analyzes logs automatically and clusters them into templates. This gives an at-a-glance view of anomalies and parameters that stand out. Users can visualize these simply by clicking on a parameter of interest.

• Anomalies: different classes of anomalies point to different errors, e.g., “flow anomalies” are used to spot issues in expected sequences of logs that show missing logs.

Coralogix and Heroku Integration

One of the reasons for choosing Coralogix for the project was the deep integration of Coralogix with Heroku. The ‘out-of-the-box’ syncing between Heroku and Coralogix provides smooth scaling, easy configuration, and proactive monitoring. Configuration in Coralogix is automatic. Once integrated with Heroku a ‘sub-system’ is created in Coralogix. This deep integration facilitates: 

Parsing logs: Coralogix is agnostic to data format. However, in this project, unstructured logs were parsed by Coralogix to extract structured data to allow for easier querying. 

Alerts: Real-time verification on the accuracy of an alert can be easily configured. A threshold can be set, and rules used to send out alerts on an ‘If This, Then That’ basis; the threshold adding a layer of granularity to the rule to optimize the output.

Machine learning-based anomalies: When Coralogix ML detects an anomalous flow in log volume an email alert is sent. This email provides the user with an interface to drill down into more detail on the anomalous event. This detail includes insight into the logs that were created during the event.

The CI/CD Pipeline in Heroku

The CI/CD pipeline process is optimized via Coralogix integration with Heroku. Coralogix machine learning capability is then used to spot trends, patterns, and anomalies. These data being mapped to a deployed version. In addition, anomalies can be tied to a specific upgrade. Coralogix is automatically added to the Heroku pipeline. Tags are used to benchmark releases, subsequently applied to analyze data across the software development lifecycle (SDLC).

How Coralogix Provides Multi-Cloud Observability 

Because of the rich functionality and deep integration capabilities of the collaboration between the team partners, the project moved in a consistent and focused manner. 

The project was expedited by:

• Coralogix unifying data across multiple data sources entering the extended platform.
• Anomalies detected by Coralogix. This reduced time-to-resolution during critical phases of development and trail runs. 
• Coralogix simplified the proactive monitoring of seemingly inconsistent types of applications.
• Coralogix working from build to run. Coralogix was able to easily work at scale and unify the platform during rollout. Because Coralogix was able to make the switch from a build focus (identify defects) to a run focus, dynamic monitoring was provided. ML capabilities are used to detect anomalous flow in the daily life of the system.

All of these features were used to optimize and inform successful deployment choices.

Metrics are expected to be deployed in the coming weeks to enhance long term deviation monitoring. 

Coralogix prides itself on the versatility of the Coralogix platform. Coupled with deep integration capability with Heroku, the team delivered a specialized and highly collaborative offering that made the project a success.