Slow Build Pipeline? Build Faster by Building Only What You Need

In today’s fast-paced software world, organizations need to deliver fast, as fast as multiple times per day. Agile methodologies and DevOps culture have contributed to enabling this constant delivery. CI/CD, Continuous Integration and Continuous Delivery, is almost a basic requirement for any company that wants to remain competitive in the market.

CI/CD allows the delivery of code changes more frequently and reliably by automating the required steps to take a working piece of software to a production environment. This automation comprises a set of steps to compile, build, and deploy code, also called a “building pipeline.” 

A fast and efficient building pipeline is fundamental for an organization to successfully practice CI/CD. However, a problematic building pipeline can slow down teams and hinder progress.

In this article, we’ll analyze the components of a building pipeline and give you actionable tips on how you can improve yours. We’ll also look at a real-world example that shows us the importance of optimizing for success.

First, let’s clarify some key terms. In Kristian Erbou’s book, Build Better Software: How to Improve Digital Product Quality and Organizational Performace, the DevOps expert defines the elements of a build pipeline: 

Pipeline: the sequence of activities you execute deterministically, one activity at a time, in the form of a workflow configured in your Continuous Integration and Delivery platform.

In software development, there are two kinds of pipelines: a build pipeline and a deployment or release pipeline. 

The build pipeline is the first step and its outcome will be part of the deployment pipeline. Keep in mind that this is a generic, language-agnostic process. Here is a visual of the generic build pipeline:

Therefore, a build pipeline is a set of activities that you perform to build an artifact.

Build: the set of activities executed on a virtual assembly line producing

the bill of materials that will eventually be part of the output in the form

of an artifact.

Artifact: a folder or a zip file containing files needed during a deployment of software to an environment. It contains the bill of materials for your upcoming deployment to nodes in your infrastructure.

The newly created artifact will then feed the deployment or release pipeline. Here is a visual of the generic deployment pipeline:

Deployment: the act of changing state on nodes in your infrastructure, like changing a file with a newer version of the same file. The experience of the end-user does not change though.

Release: A release, however, is activating a change in functionality for an end user.

Though it’s important to deploy at every commit to stay Agile and competitive. But one of the most common complaints around CI/CD is that it’s a slow process. Having a slow build pipeline can compromise your organization’s success in several ways:

1. Every extra minute you spend on these activities costs you money, from developers’ salaries to machine capacity — it all adds up.
2. Slow build pipelines decrease development cadence and break the flow state of your engineers.
3. A slow and faulty process can also take a toll on your team’s motivation and mental health.

But these issues can be fixed. If you have a slow build pipeline, there are actions you can take to correct the course and fully benefit from CI/CD and DevOps.

Diagnosis

Before you take any concrete measures, it’s important to run a diagnosis of your current status. The diagnosis is similar to any other performance-related task. 

1. Establish a way to measure your CI/CD pipeline so you can make informed decisions instead of relying on feelings and impressions.
2. Once you have your measurements, define the ideal goals for these metrics. For example: how long should your builds take? 
3. Put concrete measures into place, as we detail below. 

It’s important to regularly check your metrics against your goals. This way, you can adjust as needed without ending up too far off from your ideal situation.

A build should be able to run from start to finish in less than ten minutes—if not, you should optimize the build. Let’s look at some concrete ways in which you speed up your builds.

1. Analyze Your Pipeline

Analyze what your pipeline is doing and where time is being spent. Map it out against the value each step provides. This should give you a high-level understanding of your pipeline and answer questions such as:

• Are you spending time on high-value steps? 
• Are there any long steps that don’t bring value and can be eliminated?

Your goal is to make things simpler. For each pipeline component, decide if you should keep it or let it go. If it stays, think of ways in which you can optimize it. If you’re running into issues, execute a Root Cause Analysis.

2. Test 

Regularly execute automated tests and evaluate the tests that you’re implementing. Delete obsolete tests can simplify your codebase and give you an opportunity to remove legacy code and refocus on refactoring code in use. 

Also, test your tools. Some are faster than others, and the settings you define will impact your build time.

3. Use Caching

Downloading modules at build time takes significant time and might be a reason your build is taking so long. Regardless of the tools you use, dependencies tend to slow down the process. 

Size matters. Large artifacts can take a long time ot move from one datacenter to another. It also takes a long time to build artifacts and convert them to a compressed file, just as it takes a long time to unzip them once you’ve downloaded an artifact on a deployment target node. 

As a workaround, use caching to speed things up instead of building from scratch every time.

*Remember to not cache everything forever. Some parts will become obsolete eventually, so it’s good to clear them when required.

4. Run Jobs in Parallel

The previous tips are easy wins that cost you close to nothing. This one might require more investment if you don’t have the resources up front, but it can pay off in the long term.

To run several jobs in parallel, you might need more machines and, potentially, an upgraded license with your building tool. This means more costs, but the speed you gain can make up for it — it’s a matter of doing a cost-benefit analysis for your specific situation. Faster builds or parallel builds can lead to quicker development and delivery.

5. Build Microservices

Microservices are increasingly becoming the way to go, but they’re still not the reality at many companies. If they’re at your organization, great! You’re saving considerable build time.

But if you’re stuck with monolithic applications that take long to compile and test, build only the modules that are relevant for that commit.

6. Use Mockups

Use mockups to simulate connections to third-party services. Using real services is redundant, as you are testing your own code, not the services. Not only does this save time, but it also requires fewer resources to run and administer. 

7. Split Up Your Build

Have a strategy for dependency management and split up monolithic builds into smaller ones if quick wins are hard won in the existing pipeline. 

In Build Better Software, Erbou suggests one way to speed up your build and deploy cycletime is to have one build pipeline capable of quickly building sourcecode you can change often and one pipeline building an artifact containing the subset of the whole delivery that rarely changes.

Etsy, a global online marketplace focused on handmade or vintage items, is well known in the technology world for having mastered DevOps practices. 

During its first years, the company struggled with slow, troublesome updates that often brought down the website. This led to loss of revenue, unhappy customers, and sellers potentially taking their business elsewhere.

A new technical management team succeeded in shifting things around. From 2 deployments per week, each taking up to 4 hours, Etsy rose to as much as 50 deployments a day, with fewer disruptions. 

A fully automated pipeline is at the heart of this successful transition, along with other DevOps tools and processes, such as developer virtual machines, one-click deployments, thorough application and system monitoring, and on-call policies for dev and ops teams.

What’s remarkable in this transformation is that Etsy does not have a formal DevOps team, relying instead on a tight collaboration across teams. The website has around 60 million visits and 1.5 billion page views per month.

Increasing the speed of your build pipeline is not a one-size-fits-all approach, as it heavily depends on where you’re starting from. Some companies don’t even have automation in place, so simply achieving it is already a huge win. For others, the improvement can be about shortening the build time or even doing the same with fewer resources. 

Furthermore, not all the steps detailed above are mandatory for every case. You might get some quick wins by implementing only some of them, as it might already translate into a real improvement on your systems and your teams’ time and resources.

Most importantly, strive to achieve reliable build pipelines and focus on building only what you need. It will pay off on the long term, and you will be on your way to happy teams and happy customers.

Søren Pedersen

Co-founder of Buildingbettersoftware and Agile Leadership Coach