Tag: automation

Analyzing Test Results Through Your Logs & How to Choose Which Automation Tests to Implement

Posted on by eugene evdokimov

According to the 2021 test automation report, more than 40% of companies want to expand and invest their resources in test automation. While this doesn’t mean manual testing is going away, there is an increased interest in automation from an ROI perspective – both in terms of money and time. 

After all, we can agree that writing and running those unit test cases are boring. A good automation strategy can free up the tester’s time to tackle some of the more complex problems and help monitor logs with the early detection of bugs.

However, teams often rush to automate tests without a proper testing strategy, which causes things to break when there is an overhaul. Maximize your energy and return on investment by choosing the right tests to automate.

What should you consider before automating any test case?

1. Test Frequency

Writing that one manual test for an edge-case component is usually more efficient. Testing a new functionality allows you to learn more about the application quickly. However, that’s not effective as the number of functionalities increase.

Split your test scenarios into two sections: Ones that are repetitive and ones that are one-off or pretty complex.

Automate the ones that repeat the most. You can even set a threshold for the test frequency above, which you will consider automating.

For example, application login or alert system tests are ideal candidates for test automation since they must run after every application build. 

There are a couple of exceptions to this rule, too – say, the volume of data entry a single test needs to execute is pretty significant. In that case, it makes sense to automate that specific test since it would save a lot of time.

The only caveat here is automating a series of repetitive tests that depend on each other. In case of a failure, it can be hard to pinpoint the exact test that is the main culprit. That’s where logs come in handy and help you efficiently detect these long-term pattern failures.

2. Test Coverage

Test coverage is crucial for software quality and ensuring the stability of your software build. Automating the right tests can aid your goal of achieving high test coverage with almost the same amount of time invested. 

For example, running automated tests is a great idea if your application has many components. This bypasses the off-chance of a manual error of missing a specific test and ensures that the most critical parts of your application run smoothly. You can also run those lengthy overnight tests unattended and wake up to detailed logs of why your test failed (or worked!).

3. The Results

How predictable are the results? Automation requires pre-defined inputs and outputs to produce pass and fail conditions otherwise, they might lead to false outcomes.

If you’re in the exploratory stage of testing, and your tests are ad-hoc or require precise domain knowledge, automating them is not the best idea.

4. Feature Importance

If a project is an important feature that could disrupt the user experience if it fails, you should write an automated test suite. This way, you can safeguard against human errors messing up your release.

Ideally, the tests should be run on a continuous cycle so that relevant teams can be alerted as soon as possible.

5. Time-Reward Ratio

While automation frees up the tester’s time, organizations and individuals often overlook a crucial aspect of testing – the cost and time required to maintain the automated tests. If there are significant changes to the backend of your application, often writing and rewriting the code for automated tests is just as cumbersome as manual testing.

One interesting way to tackle this is for test engineers to automate just enough to understand which part of the program is failing. You can automate the broader application tests so that if something breaks, you know exactly where to look. Smart test execution, one of the top trends in the test automation space, does precisely this by identifying the specific tests that need to be executed.

6. Human Involvement

How complex is the test suite you’re trying to automate? If the test results need to be rechecked with a human eye or need to have actual user interaction, automating it probably won’t help a lot.

For example, user experience tests are best left unautomated because a testing software can never mimic human emotion while using a product. However, if you need visual confirmation on the test output, automated tests that take screenshots can run and later be manually verified.

7. Priority

When do you need the test results? If automating the tests helps you achieve a faster time to market the product, you should go ahead with it. However, don’t let writing and running automated tests be a bottleneck when you need the results immediately. 

Also, you should remember that ‘tests’ are not the only thing that can be automated to make your application more efficient. Tasks such as manual data gathering or setting up data inputs also make great candidates for automation. So, if there’s a large dataset but you’re running low on time, automating it might be your savior!

Frequently Automated Test Cases

1. Performance tests (Load, stress tests)

Load tests are almost notorious for being the “overnight” tests. By definition, load tests require a lot of resources simply because they identify system lag and performance issues that arise when a company is scaling up. 

That’s why tools that do automated testing make a lot of sense – because they can effectively simulate users and resources at a fraction of the cost. I mean, try finding 1,000 users to do bug testing on a product that hasn’t been released yet – yikes! 

While you definitely can’t hire 1000 QA specialists to do automation testing, a testing automation framework can set up dummy users and have them interact with your product just as a real user would. This will enable you to scale and avoid outages by identifying them early in the process. Your team can then look into the performance metrics and determine the exact cause for the loss of speed or outage.

Similarly, if you need to do cross-browser testing, automated testing can help you gather your application’s performance across multiple configurations with just a few steps.

Automate your performance tests to see where things are breaking, and whether your application can handle those breaking points.

2. Unit Tests

If you are developing the codebase of a large application, automating the unit tests will save you time. Automating testing for unit tests will help you find errors in real-time by giving you a continuous overview of whether the individual components are working or not. 

Automation is especially useful when you refactor your code since as long as the unit tests are green, you can safely assume that the behavior of individual units of code hasn’t changed. Moreover, the reports of these tests can be made available to the entire team instantly.

3. Regression Tests (Smoke, Sanity tests)

Regression tests ensure that the application runs smoothly even if many changes are made. That means several application components need to be re-tested repeatedly. Due to this repetition, regression tests are an ideal candidate for test automation.

Automating regression tests will help you save manual resources and time, and scale much faster. Although regression tests are usually performed at the end of a software release, automating them also gives you an option to run them iteratively and continuously. This helps identify bugs in the program faster and creates rapid feedback loops, thus leading to a quicker resolution.

4. Functional Testing

Functional testing verifies if an application functions the way it should on the front-end. Although some aspects of functional testing are manual, a lot of it should be automated to ensure bug-free product delivery. 

For example, end-to-end testing automation ensures key predefined user experience flows run smoothly for products with daily releases.

Using selenium to automate functional testing is a popular choice. You can even tweak the tests with a slightly different data set or user behavior to cover multiple use cases.

Which tests should you definitely not automate?

1. Exploratory testing

Exploratory tests include a wider range of essential unscripted tests that are done on the fly. Typically, these tests require some domain knowledge and familiarity with the application to determine unexpected behavior. Since they are not very well defined, they cannot be automated.

Once a tester discovers a defect through exploratory testing though, those test actions can be documented and automated for future builds. 

2. Usability Tests

As mentioned earlier, usability tests shouldn’t be automated because it’s hard to predict human behavior. This could range from wrong fonts, colors, or a UI that confuses people. You’ll only know these when you perform either beta or QA testing. Although there are tools that try to automate this, it is more effective (and less expensive) to get a human to look at it.

To Automate or Not To Automate

Test automation is crucial for an efficient CI/CD pipeline. A lot of innovation has been going on in the test automation space, such as parallel test execution, DevTestOps, IoT test automation, etc. These automation frameworks have helped drastically reduce the time to market for products and increase the build quality.  

Choosing the right test to automate is just the first step to achieving that for your organization, so test faster, fail faster, and fix faster!

Intro to AIOps: Leveraging AI and Machine Learning in DevOps

Posted on by eugene evdokimov

AIOps is a DevOps strategy that brings the power of machine learning to bear on observability and system management. It’s not surprising that an increasing number of companies are now adopting this approach.  

AIOps first came onto the scene in 2015 (coincidentally the same year as Coralogix) and has been gaining momentum for the past half-decade. In this post, we’ll talk about what AIOps is, and why a business might want to use it for their log analytics.

AIOps Explained

AIOps reaps the benefits of fantastic advances in AI and machine learning in recent decades.  Because enterprise applications are complex, yet predictable systems, AI and machine learning can be used with great effect to analyze their data and extract patterns. The AIOps Manifesto spells out five dimensions of AIOps

  1. Data set selection – machine learning algorithms can parse vast quantities of noisy data and provide Ops teams with a curated sample of clean data.  It’s then much easier to extract trustworthy insights and make effective business decisions.
  2. Pattern discovery – this generally occurs after a data set has been appropriately curated. It involves using a variety of ML techniques to extract patterns. This can be rule-based or neural networks that involve supervised and unsupervised learning.
  3. Inference – AIOps uses a range of inference algorithms to draw conclusions from patterns found in the data. These algorithms can make causal inferences about system processes ‘behind the data.’  Combining expert systems with pattern-matching neural networks creates highly effective inference engines.
  4. Communication – For AIOps to be of value it’s not enough for the AI to have the knowledge, it needs to be able to explain findings to a human engineer! AIOps has a variety of strategies for doing this including visualization and natural language summaries.
  5. Automation – AIOps achieves its power by automating problem-solving and operational decisions. Because modern IT systems are so complex and fast-changing, automated systems need to be intelligent. They need machine learning to respond to quickly changing conditions in an adaptive fashion.

Why IT needs AIOps

As IT has advanced, it has shouldered more and more of the essential processes of business organizations.  Not only has technology become more sophisticated, it has also woven itself into business practice in increasingly intricate ways.

The ‘IT department’ of the ‘90s, responsible for a few niche business applications, has virtually gone. 21st century IT lives in the cloud. Enterprise applications are virtual, consisting of thousands of ephemeral components.  Businesses are so dependent on them that many business processes are IT processes.

This means that DevOps has had to upgrade. Automation is essential to managing the fast-changing complexity of modern IT. AIOps is an idea whose time has come. 

How companies are using AIOps

Over the past decade, AIOps has been adopted by many organizations. In a recent survey, OpsRamps found that 68% of surveyed businesses were experimenting with AIOps due to its potential to eliminate manual labor and extract data insights.

William Hill, COTY, and KPN are three companies that have chosen the way of AIOps and their experience makes fascinating reading:

AIOps Case Study: William Hill

William Hill started using AIOps to combat game and bonus abuse. As a betting and gaming company, their revenues depended on people playing by the rules and with so many customers, a human couldn’t keep track of the data.

William Hill’s head of Capacity and Monitoring Engineering, Andrew Longmuir explains the benefits of adopting AIOps.  First, it helped with automation, and in particular what Andrew calls “silo-busting”. AI and machine learning allowed William Hill to integrate nonstandard data sources into their toolchain.

Andrew uses the analogy of a jigsaw. Unintegrated data sources are like missing pieces of a puzzle. Using machine learning allows William Hill to bring them back into the fold and create a complete picture of the system.

Second, AIOps enables William Hill’s team to solve problems faster.  Machine learning can be used to window data streams, reducing alert volumes, and eliminating operational noise.  It can also detect correlations between alerts, helping the team prevent problems before they arise.

Finally, incorporating AI and Machine Learning into William Hill’s IT strategy has even improved their customer experience. This results from them leveraging insights extracted from their analytics data to improve the design of their website.

Andrew has some words of wisdom for other organizations considering AIOps. He recommends focusing on a use case that is central to your company.  Teams need to be willing to trial multiple different solutions to find the optimum setup.

AIOps Case Study: COTY

COTY adopted AIOps to take the agility and scalability of their IT strategy to the next level. COTY is a major player in the cosmetics space, with clients that include Max Factor and Calvin Klein.  As a dynamic business, they relied on flawless and versatile performance from their IT infrastructure to manage everything from payrolls to wireless networks.

With over 4,000 servers and a cloud-based infrastructure, COTY’s IT system is far too complex for traditional DevOps strategies to handle. To deal with it they’ve chosen AIOps.

AIOps has improved the way COTY handles and analyzes data. Data sources are integrated into a ‘data lake’, and machine learning algorithms can crunch its contents to extract patterns.

This has allowed them to minimize noise, so their operations department isn’t bombarded with irrelevant and untrustworthy information. 

AIOps has transformed the way COTY’s DevOps team thinks about visibility. Instead of a traditional events-based model, they now use a global, service-orientated model.  This allows the team to analyze their business and IT holistically.

COTY’s Enterprise Management Architect, Dan Ellsweig, wants to take things further. Dan is using his AIOps toolchain to create a dashboard for executives to view. For example, the dashboard might show the CTO what issues are being dealt with at a particular point in time.

AIOps Case Study: KPN

KPN is a Dutch telecoms business with operating experience in many European countries.  They adopted AIOps because the amount of data they were required to process was more than a human could handle.

KPN’s Chief Product Owner Software Tooling, Arnold Hoogerwerf, explains the benefits of using AIOps. First, leveraging AI and machine learning can increase automation and reduce operational complexity. This means that KPN’s DevOps team can do more with the same number of people.

Secondly, AI and machine learning can speed up the process of investigating problems. With traditional strategies, it may take weeks or months to investigate a problem and find the root cause. The capacity of AI tools to correlate multiple data sources allows the team to make crucial links in days that otherwise would have taken weeks.

Finally, Hoogerwerf has a philosophical reason for using AIOps.  He believes that while data is important, it’s even more important to keep sight of what’s going on behind the data.

Data on its own is meaningless if you don’t have the knowledge and wisdom with which to interpret it.

Implementing AIOps with Coralogix

Although the three companies we’ve looked at are much larger than the average business, AIOps is not just for big companies. The increasing number of platforms and vendors supporting AIOps tooling means that any business can take advantage of what AIOps has to offer.

The Coralogix platform launched two years after the birth of AIOps and our philosophy has always paralleled the principles of AIOps.  As Coralogix’s CEO Ariel Assaraf explains, organizations are burdened with the need to analyze increasing quantities of data. They often can’t do this with existing infrastructure, resulting in more than 99% of data remaining completely untapped.

In this context, the Coralogix platform is a game-changer. It allows organizations to analyze data without relying on storage or indexing. This enables significant cost savings and greater data coverage. Adding machine learning capabilities on top of that makes Coralogix much more powerful than any alternative in the market. Instead of cherry-picking data to analyze, stateful stream analysis occurs in real-time.  

How Coralogix can help with pattern discovery

One of the five dimensions of AIOps is pattern discovery. Due to the ability of machine learning to analyze large quantities of data, the Coralogix platform is tailor-made for discovering patterns in logs. As a case in point, gaming company AGS uses Coralogix to analyze 100 million logs a day.

The patterns extracted have allowed their DevOps team to reduce MTTR by 70% and their development team to create enhanced user experiences that have tripled their user base.

Another case is the neural science and ML company Biocatch. With exponentially increasing log volumes, their plight was a vivid illustration of the complexity that 21st century DevOps teams increasingly face.

Coralogix could handle these logs by clustering entries into patterns and finding connections between them. This allowed Biocatch to handle bugs and solve problems much faster than before.

How Coralogix can communicate insights

Once patterns have been extracted, DevOps engineers receive automated insights and alerts about anomalies in the system behavior.  Coralogix achieves this by integrating with a variety of dashboards and visualization solutions such as Prometheus and CloudWatch.

Coralogix also implements a smarter alerting system that flags anomalies to DevOps engineers in real time.  Conventional alerting systems require DevOps engineers to set alerting thresholds manually. However, as we saw at the start of this article, modern IT is too complex and fast-changing for this approach to work.

Coralogix solves this with dynamic alerts. These use machine learning to adjust thresholds in response to data.  This enables a much more effective approach to anomaly detection, one that is tailored to the DevOps landscape of the 21st century.

Wrapping Up

The increasing complexity and volumes of data faced by modern DevOps teams mean that humans can no longer handle IT operations without help.  AIOps aims to leverage AI and machine learning with a view to converting high-volume data streams into insights that human engineers can act on.

AIOps fits with Coralogix’s own approach to DevOps, which is to use machine learning to help organizations effectively use the increasing volumes of data they generate.  Observability should be for the many, not just a few.

Test Automation Tools to Accelerate CI/CD

Posted on by eugene evdokimov

So much of our world has moved away from the slow and methodical, towards the agile and iterative. In transport, for example, everything is “on demand”, constantly changing and adaptable. The same is true for developers. With movements and philosophies such as CI/CD solutions, everything is about moving quickly, yet smartly.

Test automation is an integral part of this development philosophy. We’ll take a look at 3 popular test automation tools, and help you decide which one is the right fit for your organization.

The Importance of Test Automation

Before we jump into the 2 testing tools, a quick note on the importance of test automation. Automated testing frameworks can “help quality assurance engineers define, execute, and automate various types of tests that can help development teams know whether a software build passes or fails.” In the CI/CD environment, test automation literally becomes part of the continuous flow and cycles of development.

Let’s take a look at the 3 platforms:

Test.ai

Test.ai is a machine learning-based platform that has been trained to recognize common user scenarios and execute test cases. The platform leverages AI to identify the various screens and elements in your app, and to execute user scenarios to enable you to test on-demand whenever you’re ready. It also recognizes elements, so that things don’t break even when they change.

Test.ai is relatively quick and easy to set up, requires no programming knowledge for more basic functions, and the platform can execute tests even when the user interface or certain flows in the app change.

Even though the company has been around since October 2015, and has raised around $17m, there is a definite lack of quality peer reviews available. This is concerning as most companies of this size would have multiple reviews available, from different companies and roles, giving an indication of the level of robustness of the solution, as well as insights into which types of projects the platform is better suited to.

Test.ai claim “larger unnamed partners that ‘make app stores’ or devices…working at the stratospheric level having to verify tens of thousands of apps to ensure that everything is in working order”.

Testcraft.ai

Testcraft bills itself as “codeless Selenium with ai maintenance”, for automated manual testing with no framework or maintenance. It is primarily a platform for web apps, and is a complete SaaS solution, with a shallow learning curve (and no coding required).

Its automation platform includes regression and continuous testing capabilities, a drag-and-drop interface, and the ability to run tests on multiple browsers and work environments at the same time. TestCraft promises faster test creation, execution, and maintenance, through its dynamic test model that can be updated to reflect changes to your app. Its interface is simple and easy to use.

Areas, where there could be an improvement, are an occasional lack of responsiveness, some issues with urls from outside of the app itself, changing datasets can be confusing, and there are quite a few bugs that still need to be ironed out.

Testim.io

Testim.io leverages machine learning for the authoring, execution, and maintenance of automated test cases. The platform uses dynamic locators and learns with every execution. The outcome is super fast authoring and stable tests that actually learn and improve, eliminating the need to continually maintain tests with every code change.

Well known companies such as Netapp, Verizon Wireless, Wix.com, and others run over 500,000 tests every month using Testim.io. The platform has gained acclaim for significantly shortening the time from development to production, to minutes in some cases. It allows the testing of new features with full regression test coverage. Test stability is known to be high, with easy-to-understand results. The platform is also easy to connect to your chosen CI and discover and fix bugs, with helpful screenshots along the way. Customer support is known to be excellent.

But, there are some issues that have been brought up for improvement, such as image verification, but these are being dealt with according to company communications, and barely detract from what is otherwise an excellent product.

Comparison

Most reviews conclude with some form of a “we cannot recommend one product, it all depends on your needs”. In this case, however, we have found testim.io to be a top performer in most categories, and feel compelled to recommend it.

This choice is further enhanced by the numerous positive reviews and big names using the platform.

Continuous Improvement

When it comes to continuous integration and continuous delivery, areas like automated testing and automated log insights become essential, integrated tools – not merely welcome additions.

For machine learning-powered log analytics, look no further than Coralogix. With Coralogix, you get virtually endless seamless integrations, dashboards and live streams, view hours of data in seconds and get automated insights into any log issues. CI/CD has found the perfect log partner.