How to Implement Effective DevOps Change Management

A decade ago, DevOps teams were slow, lumbering behemoths with little automation and lots of manual review processes. As explained in the 2020 State of DevOps Report, new software releases were rare but required all hands on deck. Now, DevOps teams embrace Agile workflows and automation.  They release often, with relatively few changes.  High-quality DevOps change management is no longer a nice-to-have, it’s a must. 

For a lot of DevOps teams, this is easier said than done. For the State of DevOps Report, over 2000 DevOps professionals were surveyed on topics including the implementation of change management.

The survey findings uncovered the existence of four main change management styles stark contrasts in terms of characteristics and effectiveness. Successful DevOps teams were three times more likely than others to have superior change management.

We’ve divided those styles into the good, the bad, and the ugly. Let’s take a look.

The Good

 First, we’ll look at two approaches that both exemplify high-quality change management by putting the emphasis on engineering.

Engineering-Focused

DevOps teams at cutting edge technology companies tend to implement engineering-driven change management. The stand-out feature of this strategy is that it leverages automation to facilitate a rapid deployment cycle.

Changes as Code

Rather than using traditional documentation to record software changes, engineering-driven DevOps tries to express as much as possible in code.

For example, infrastructure as code enables DevOps teams to apply the same principles to application infrastructure that they would to software. They can take full advantage of Git-style versioning, central repositories to push their changes to, and Continuous Integration.

As well as harnessing the full power of automation in the CI/CD cycle, changes as code allow DevOps teams to sidestep ITIL review processes. Because traditional review processes are manual and labor-intensive, cutting them out of the loop is like letting the brakes off CI/CD.

This investment in automation pays off.  The State of DevOps Report found that 75% of companies surveyed had efficiency that was either high or very high and 67% had high or very high implementation success.

Operationally Mature

DevOps teams at large, established companies face a different set of challenges than those at smaller startup companies. These teams often work in traditional sectors, like energy, which came relatively late to the digital party.

Because these sectors have a mature customer base and strong regulatory requirements, DevOps teams must strike a balance between managing rapid change and taking as few risks as possible. High-levels of automation offer operationally-mature change management, which is often complemented by heavy use of manual, established review processes.

The State of DevOps found that while operationally mature companies score low in efficiency (68% had very low efficiency), they score much higher in performance sentiment. 73% of respondents had performance sentiment that was high or very high.

While traditional approval processes cause reductions in efficiency, it’s generally considered necessary in tightly regulated sectors such as finance. Later we’ll see how automated platforms are designed to ensure ‘Continuous Compliance’, meaning the work of an approval committee can now be done with code. 

The Bad

Ad Hoc

This is how not to do change management. Ad hoc change management is typically used by DevOps teams in small companies that haven’t quite ‘got with the program’ of 21st century DevOps. In contrast to engineering-focused companies, they have very little automation. And in contrast to operationally mature companies, they have few standardized approval processes.

These shortcomings, both of automation and formal processes, leads to a low performance sentiment. The State of DevOps Report found that 80% of respondents doing ad hoc change management had low or very low performance sentiment.

The Ugly…

Governance-Focused

This approach tends to be used at large companies that have large revenues and employee head counts.  The report found that a third of companies surveyed had revenues above $1 billion. Nearly half had over 5,000 staff on their payroll.

As the name implies, governance-focused change management puts the majority of the emphasis on traditional approval and manual review processes.  Automation is often deemphasized.  This results in inefficiency and low morale. The State of DevOps Report found that 70% of respondents had low or very low performance sentiment and 62% had low or very low implementation success.

The Trouble with Toil

Change management in DevOps is a forest full of traps. One of these traps is toil. Identified by Googler Vivek Rau, toil is “the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.”

Examples of toil can include responding to pager alerts and manually running a script every time you want to deploy to a cluster.  Toil is work that has to be done, again and again, simply to keep a service up and running.

In many ways, toil is a degenerate form of change management. It’s reactive, involving little strategy. It doesn’t add value, tending to leave the system in the same state it was in before. Typically, toil results from poor system design and lack of appropriate automation.

The State of DevOps Report found that companies employing each of the four DevOps change management styles had different levels of toil. 

Engineering-focused companies had the lowest toil, a statistic that makes sense given their commitment to automation. These were followed by ad hoc and governance-focused DevOps teams. The worst offenders were operationally mature companies. 63% of respondents admitted that toil took up 30% or more of their time.  This correlates with their relatively high emphasis on manual review and traditional approval processes.

Toil is detrimental to healthy DevOps culture because it takes away from engineering time. Because toil grows linearly with service size, there’s the danger that it could end up taking 100% of DevOps engineers’ work hours. Not only is this detrimental for mission-critical workloads, but this could also sap DevOps team morale and discourage new hires from permanently joining a team.

Smart Change Management with Automation

The best way to minimize toil is the intelligent use of automation platforms.  Google is one of the leaders in automating DevOps change management.  The size of their system (worldwide) coupled with the sheer volume of traffic and potential scope for failure leads them to develop unique solutions such as Borg for MySQL.

These days, DevOps teams have many choices when it comes to DevOps automation platforms. Below are two of the most popular platforms.

Puppet

With customers including RBS and Staples, Puppet is the industry standard for automation in DevOps change management. Puppet possesses many solutions tailored to support a DevOps team’s needs.

Puppet Enterprise is an integrated DevOps platform that can handle a range of change management processes such as Continuous Delivery, Patch Management, and Continuous Compliance.

Puppet Forge enables teams to automate DevOps workflows and innovate quickly through the use of standardized components. It’s essentially a communal repository of Puppet modules. Each module performs a specific function, such as installing MySQL. Teams can use these modules as building blocks to create customized DevOps procedures. This allows engineers to bring the powerful principles of code reuse and DRY to DevOps. 

Bolt is an open-source solution that is optimized for automating DevOps workflows. With it, teams can update their system, troubleshoot servers, and deploy applications.

Chef

Chef is used by customers like IBM, Facebook, and Capital One. It’s adapted for use in a highly regulated DevOps environment. This makes it particularly useful for large operationally mature and governance-focused companies, such as those in the financial sector.

The essence of Chef is its Chef Enterprise Automation Stack, a set of technologies that automates a company’s DevOps processes. It contains several platforms, each tailored for a different facet of DevOps change management. Let’s look at two of them.

Chef Infra automates infrastructure management by implementing the infrastructure as code paradigm. This makes your application easily scalable and portable as well as encouraging best practice such as Test-Driven Development. Security is baked into Chef Infra through its use of an agent-based approach, making it attractive for financial organizations.

Chef Habitat handles the configuration, management, and behavior of your app.  It’s technology agnostic, so developers don’t have to worry about infrastructural details when deploying new releases. It’s well-suited to applications with plenty of dependencies and frequent updating requirements.

People Power and DevOps Change Management

While automation is important, we can’t lose sight of the crucial role played by a well-functioning DevOps team. According to the State of DevOps Report, organizations that keep their employees in the change management loop are five times more effective than those that don’t.

 Automation requires employee involvement. Organizations implementing automation need to critically involve their team members through the use of feedback loops.

Wrapping Up

Automation is essential to innovation in DevOps change management. As 2020’s State of DevOps Report points out, it builds team confidence in the change management process and minimizes toil, allowing developers to focus on creative engineering projects. 

Whatever the size of your DevOps team or organization, embracing automation and keeping your personnel in the loop is the key to change management success.