CI/CD – What You Need to Know

Continuous integration (CI) and continuous delivery or deployment (CD) cover the process of automatically merging, building, and testing code changes ready for release, and – in the case of continuous integration solutions – releasing those changes to users.

If you’re developing software for others to use, you’ll need to go through some form of build and test process before you make your latest changes available. In the past that process was often manual and very time-consuming – annual release cycles were not unusual and quarterly releases felt like frequent updates.

To give you an idea of why the process took so long – and the issues that CI/CD seeks to remedy – it helps to have a little glimpse of what software development used to look like. Typically, developers would work on new features for weeks or months, before integrating their code changes with their colleagues’ work. At that point, the team would try to build the solution to check if it worked as expected (and usually find a number of problems). 

Once they’d got the source code to build successfully, the development team would hand it over to a quality assurance team (QA) to check for defects. Those issues would either have to be addressed or documented until a future release. QA, in turn, would hand it over to a release or operations team to make it available for users and provide support for any issues that arose in production. The whole process could take months and didn’t result in a great experience for users.

Continuous integration solutions and delivery/deployment automates these build, test, and release processes as much as possible so as to improve the quality of the software being produced and get changes out to users faster and more frequently. With big names in software having led the way with multiple daily updates, and user expectations rising accordingly, an automated CI/CD pipeline is now a key practice in software development.

What is a CI/CD pipeline?

You can visualize the CI/CD process as a pipeline, with the updated codebase entering at one end and moving through a series of checks (depicted from left to right). If everything goes smoothly, the last stage is to deploy the software to production. If any of the checks fail, the code changes drop out of the pipeline (as they’re not fit for release) and circle back around to the beginning to be fixed, before starting the process again.

Shifting left

That flow from left to right also explains one of the key tenets of DevOps: shift left. If you imagine the entire process of software delivery – from capturing requirements and writing code to releasing to users – as a linear process moving from left to right, then shifting tasks to the left means addressing them earlier in that process.

By shifting integration and testing to the left you uncover any defects in the code earlier. That saves time overall because you identify and fix issues before further changes are built on top of the faulty code.

The software development lifecycle

Rather than a linear process, it’s even more accurate to see the software development process as a continuous cycle, with data from use in production – in the form of user feedback, bug reports, or log analysis – informing design and development of new functionality and enhancements. After all, by its very nature, software can be adapted and updated as we learn how it is used by real users. It makes sense that insights from use in production should feed into what we do next.

An efficient CI/CD pipeline not only supports that continuous cycle with regular releases but also enables many smaller feedback loops from regular builds and testing, which helps shift learning and feedback to the left.

What are the stages of a CI/CD pipeline?

While the details of a given CI/CD pipeline will vary according to organizational requirements and the product or service being developed, the overall process will include building, testing, and deploying, with testing taking the bulk of the time and effort.

As we’ve discussed elsewhere, automating your testing activities not only saves time but also helps to ensure consistency. It’s good practice to break testing down into multiple layers, with more coverage at lower levels where tests are quicker to write and maintain. Smaller tests are quicker to run and therefore provide faster feedback, so running these first gives you the biggest return. Once you’ve established some confidence in the latest code changes, you can then invest in more involved testing.

A typical project might start with a wide layer of unit tests that are run before the build. If these pass, a selection of smoke tests are run to sanity check that the basic functionality still works. You might then run integration and contract tests on the new build before deploying it to a staging or preview environment for automated end-to-end tests and any manual tests.

With CI/CD being a widely adopted practice, there is a range of open-source and third-party tools to choose from to help coordinate the stages, trigger scripts, and report on outcomes.

What does continuous integration involve?

The first part of the pipeline is continuous integration. CI seeks to avoid the pain of integrating months’ worth of work from multiple developers in one go by requiring everyone to commit their changes to a shared branch regularly. Regular commits ensure that everyone is building on the same foundation and make it possible to build and test the entire project on a regular basis.

Before you can implement CI, it’s essential that you have all your code in a source control system (although these days it’s more unusual to find a project that isn’t in source control). With version control in place, you can get started by setting up a CI server and triggering a build on each commit to the mainline or a particular development branch (according to your chosen branching strategy).

If you already have some automated tests, you can also trigger these to run automatically. If not, this is a good time to start building your automated test coverage, starting with low-level tests.

What’s the difference between continuous delivery and continuous deployment?

Once you’ve put your application or system through its paces with a barrage of automated tests, the next stages involve deploying your changes to pre-live environments and finally releasing them to users. Deployment to a testing or staging environment might be required for some automated tests (such as browser, platform, or performance tests) in which case the boundary between CI and CD is somewhat blurred.

What you do after all your automated tests have been completed successfully determines whether you’re practicing continuous delivery or deployment. With continuous delivery, changes are deployed to pre-live environments automatically but the final release to production requires manual intervention. By contrast, with continuous deployment there is no manual check; each change that successfully passes all previous stages is automatically deployed to live.

Continuous deployment works well for webpages and web apps where users don’t need to be informed of a new version or update. If there is no need to limit how often you release changes, you can update your software incrementally each time a change passes your quality assurance process. With only the latest version of your software in production, you can use a simple branching strategy with the head of your mainline always matching what’s released.

Continuous delivery, with its manual step for deploying changes, works well if you need a manual testing or verification phase, or if you need to batch changes up into versioned releases (for example, for an installed application).

If you’re practicing continuous delivery, it’s important to have a plan in place for addressing urgent issues in production. In some cases, it might be as simple as reverting the change, but there are always situations (such as database schema changes), which require a fix forward. With continuous deployment that’s simple enough, but if you’re queuing changes for delivery and need to be able to bypass them with a hotfix, you’ll need a branching strategy and deployment process to support that.

Where does continuous improvement fit into the CI/CD process?

Implementing CI/CD is an ongoing process that can involve cultural changes as well as technical challenges. It lends itself well to being built up incrementally and – in the spirit of agile – it’s good practice to learn as you go so that you can keep refining your process. By building observability into your CI/CD pipeline you can use log data to understand which stages are taking the most time, monitor how frequently failures arise at different points in the process, and measure how long it takes to get a fix into production.

Armed with data from your CI/CD logs you can find opportunities to parallelize tasks, identify common test failures that could be caught earlier, and make your pipeline more efficient so that when you do identify an issue in production, you can deploy a fix quickly.