We’re on the verge of something here, people.
A growing number of companies are shipping software in minutes.
Yeah, you read that right. Minutes. Not hours, not weeks, months, or longer. Minutes.
Often, teams struggle to ship software into the customer’s hands due to lack of consistency and excessive manual labor. Continuous integration (CI) and continuous delivery (CD) deliver software to a production environment with speed, safety, and reliability.
In today’s post, I’ll introduce these concepts, show you how to get it right, and identify what’s important. Then, I’ll include a list of tools commonly used to implement CICD. When you finish reading, you’ll have a better understanding not only of all the benefits that these practices bring, but also the challenges you might encounter.
Simple enough?
Let’s talk CICD.
Let me start by quoting ThoughtWorks’ definition for CI:
Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early. By integrating regularly, you can detect errors quickly, and locate them more easily.
With CI, a developer practices integrating the code changes continuously with the rest of the team. The integration happens after a “git push,” usually to a master branch—more on this later. Then, in a dedicated server, an automated process builds the application and runs a set of tests to confirm that the newest code integrates with what’s currently in the master branch.
If you’re doing CI and for some reason the integration fails, that means the broken build becomes the highest priority to fix before continuing to add more features. System quality—not just velocity—is important. CI works in three simple stages: push, test, and fix. But despite this simplicity, CI might become challenging if only a few members of the team practice it. Consequently, CI also requires a change in culture and support from management.
Next I will explain the three stages of the CI workflow.
Of all the three stages of CI, committing to master is the easiest technically, but the hardest culturally. And the reason that’s true is that integrating code daily doesn’t mean that a developer will push the code into a feature branch. No, the CI practice is about pushing code into the master branch, because that’s the branch that’s going to be used to release software. The “push to master” stage is also known as trunk-based development, and there’s a dedicated site that explains this technique in much more detail.
When you practice CI, it doesn’t mean you’ll no longer use branches. You still will. The only difference is that because you amplify feedback when you integrate code continually, branches become temporary. A branch might live only for the day; then it’s integrated into the master branch.
But what about incomplete changes? Well, you can integrate incomplete changes by using feature flags.
Feature flags are an if condition determining whether to run the new code or not. If a change isn’t complete yet, the flag is off by default. That way, when you integrate the code, the rest of the team has a chance to review it. The same technique applies if the new code has bugs.
To validate each time a developer integrates new code, CI relies on an automated and reliable suite of tests. If you need to compile the code, the first test is that the code compiles. Then, you can include as many tests as you consider critical.
How many tests should be included? To determine that, remember that CI’s purpose is to provide feedback as soon as possible. If a developer has to wait an hour to get feedback, it won’t work. A developer will prefer to wait rather than push, and the push might happen another day. Plus, if the build takes too long, it’ll require the team to coordinate who can push. See? That’s why the team might prefer to go back and use long feature branches. Keep the CI loop as short as possible. You might have to lean heavily on unit testing and lightly on integration testing.
Lastly, the suite of tests should be able to say that something is broken; tests should be reliable. You’re always going to miss things. But when you spot a bug in production, create a test case and include it in the CI loop. You can easily confirm that it’s reliable: If the flag for the bug fix is off and the build breaks; if the flag for the bug fix is on, the build works.
When the build is broken, fixing it should be the priority for the team. And the importance of fixing it should be a shared mindset in the culture.
Fixing the build, as a principle, comes from the Toyota andon cord that the car manufacturer used to produce cars. John Willis wrote a post on the subject, and in it, he describes Toyota’s process like this:
Toyota implemented the Andon Cord as a physical rope that followed the assembly line and could be pulled to stop the manufacturing line at any time. Furthermore, this wasn’t an ask permission to stop the line. The pull actually stopped the line.
In software, the equivalent of a Toyota andon cord is that every time a build is broken, no one else can continue pushing code changes.
Contrary to what you might think, at Toyota, the andon cord is pulled many times a day. Everyone on the team is aware of the issues happening. Besides, whoever breaks the build will try to fix the problem, and if help is needed, then the rest of the team is there. In software, if the build can’t be fixed within minutes, the team should decide if they’ll remove the code or turn the feature flag off. Having a “green” build is not one person’s problem anymore. It should be a priority, like it is at Toyota.
The idea behind fixing a broken build is that the build is always going to produce working code that’s okay to release.
CI is mainly a cultural shift, but some tools could help you to get the job done quickly. And you can even start on a dollar a day, according to James Shore. All you need is an old computer, a rubber chicken (really!), and a desk bell. Shore’s post is hilarious, and I recommend you read it. He makes a valid point clear: lack of tools isn’t an excuse. You can do CI without them.
Nonetheless, tools can help. Here’s a list of common tools that you can start using today.
There are more CI tools, but I wanted to keep the list short with the tools I’ve personally used. It’s hard to define my favorite tool, but at the moment I’d say that it’s Azure Pipelines. With it, you can integrate a lot of tools. Plus, it’s easy to use, and you can define pipelines as code.
Let me use the definition for continuous delivery from Jez Humble’s site:
Continuous Delivery is the ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way. We achieve all this by ensuring our code is always in a deployable state, even in the face of teams of thousands of developers making changes on a daily basis.
Ha! CD sounds fantastic, right? Indeed. You’re delivering changes of all types into a live environment all the time; you can ship configuration changes, infrastructure changes—everything! Usually, CI is known to be a developer’s practice and CD an operator’s practice. CI’s mission is to provide an artifact at some point in time of the application that satisfies customer expectations—in other words, that has good quality built in.
CI’s mission is then to move those artifacts throughout all the different environments of an organization’s development lifecycle. What’s critical in CD is that it will always deploy the same artifact in all environments. Therefore, a build in CI happens only once and not for each environment. The artifact produced will work with placeholders or environment variables for the build-once approach to work.
Another critical factor is that for a deployment to be smoother, each environment different than production should be similar. Development, testing, and staging should be a production-like environment. Homogeneous environments might be hard to achieve in big organizations, but the idea is to use the same tooling, process, and configurations in all the environments. In CD, production is not a special environment; it’s just another stage in the pipeline.
At some point, deployments to any environment will be boring. And that’s a good thing!
The following are the vital principles for continuous delivery. When you put them into practice, they’ll help you to deliver software safely and quickly in a sustainable way, as per the CD definition I included above.
As the saying goes, “if it hurts, do it more often.” Usually, every time a deployment happens, the application’s stability is at risk. Therefore, we tend to distance deployments from each other. But the problem with that approach is that we end up accumulating many changes. Chances are that one of those changes might have problems, forcing us to roll back the other changes that were working. Been there, done that.
What CD fosters—with the combination of CI, which implies quality is built in—is that we should work in small batches.
When you need to make a significant change in the application, use feature flags as Facebook did before releasing Messenger. Or apply the strangler pattern and split complicated changes into small and simple changes. If you do deployments more often and work in small batches, the risk of doing deployments will be lower.
At Toyota, Jidoka, or “automation with a human touch,” is the philosophy used for automation. All repeated manual labor that humans or engineers do all the time in every deployment should be automated. Humans following a recipe by memory or reading an instruction manual will fail when they’re under pressure. Conversely, machines are perfect candidates for repetitive tasks. Automation is a critical principle in CD because it helps to increase the sustainability of the process.
Be cautious, though. Once you introduce automation, you still need people’s minds to improve the process. You can achieve automation with the help of practices like infrastructure-as-code, pipeline-as-code, or configuration management.
CD will help to produce one-click deployments that can be triggered on demand.
If you don’t measure, you can’t improve. Moreover, if you try to implement all of what you’ve read in this post, you’ll get overwhelmed. You might decide not to do anything at all. If you know there are a lot of improvement opportunities, start with simple things. At some point, as I said before, deployments will be boring with low risk.
Are you currently compiling the application from a developer’s machine? Start by using a dedicated server for CI, and build the application in a different server. Are you manually copying and pasting application artifacts? Start by automating the copying/pasting process. It doesn’t matter if you’ve added small improvements or if you’re happy with how advanced the process is. There’s always going to be something to improve. CD doesn’t have an end date.
CD produces happier teams because now the deployment pipeline isn’t just an operations problem. Operations will seek ways to help developers build software with quality. And they’ll do it not just by coaching, but also by providing all the necessary tools that a developer might need to understand problems better. A team won’t optimize the process just for them, but it will add improvements that help the organization deliver more value to customers. That’s instead of the traditional approach of rewarding developers for how fast they ship, or rewarding operations for how reliable the system is. If that’s what you’re doing, these two teams will have different goals, and that won’t work.
Everyone in the team is responsible for creating a safer, quicker, and deterministic delivery pipeline continuously.
A few of the tools for CD are also tools for CI. That’s why I’ll repeat a few tools here from the CI tools list. But there’s also a few new ones.
Again, this list fails to capture how many tools are actually out there. But the purpose is only to give you a few options. You can use whatever tool works best for you.
Lastly, let me try to assemble all of the concepts I’ve discussed in this post with a traditional but straightforward workflow for a CI/CD pipeline using a Microsoft stack. Each piece can be interchangeable with other tools, platforms, and languages.
What’s important is the workflow, which you can see illustrated here. You’ll definitely want to take a look at the illustration so the summary below makes sense.
In summary, here’s what it shows:
When a release to another environment like test or production happens, steps 1–4 won’t need to be run again.
CI/CD helps teams to be more productive when shipping software with quality built in. But the road to having one-click deployments that you can produce on demand is not an easy one. That’s mainly because even though there are powerful tools that will help you to achieve CI/CD more effectively, CI/CD requires a cultural change: a mindset that every person in the team needs to understand very well.
I’d recommend that you continue learning with books like Continuous Delivery from Jez Humble. Try to focus more on how to adopt the principles and practices of CI/CD, rather than how or which tools to use. Lastly, CI/CD implementation doesn’t have an end date, like a project, where once you’re at a certain stage you can forget about it. It has to improve continuously. Platforms like Retrace that give you centralized logging and error tracking will help you to improve continuously based on the feedback you receive after each release to production.
Try Stackify’s free code profiler, Prefix, to write better code on your workstation. Prefix works with .NET, Java, PHP, Node.js, Ruby, and Python.
Stackify's APM tools are used by thousands of .NET, Java, PHP, Node.js, Python, & Ruby developers all over the world.
Explore Retrace's product features to learn more.
If you would like to be a guest contributor to the Stackify blog please reach out to stackify@stackify.com
