Leadership at the Edge

Agile Development and the Lean Startup methodology have demonstrated that success comes through iteration and feedback. The more iterations you have, the higher your chances of achieving success. As an engineering leader, enhancing deployment frequency—and thus increasing the number of iterations—might be the most impactful thing you can do for your company’s success.

Deployment frequency is one of the four DORA metrics listed below. It’s one of the two speed metrics, the other being Lead Time for Changes. When both are in the high performer range, the team can move very quickly, providing the business with many iterations. While Lead Time impacts the length of your feedback cycle, Deployment Frequency affects scheduling and concurrency. Focusing on Deployment Frequency can make experimentation easier for the team.

• Deployment Frequency (DF): Deployment frequency measures how frequently the team puts code into production.
• Lead Time for Changes (LT): Lead time measures how long it takes from an engineers first commit on a task until it is in production.
• Change Failure Rate (FR): Change failure rate measures the percentage of deploys that lead to failure. A failure is defined by the business, but it is usually related to support escalations and or monitors being triggered.
• Mean Time To Recover (MTTR): Mean time to recover measures how long it takes to relieve the failure. This doesn’t necessarily mean fixing the underlying issue, just allowing work to continue. Good teams restore service in less than 1 day.

The goal of this post is to provide a few tips for improving your team’s Deployment Frequency. A successful Devops transformation will involve improvement in all 4 metrics, however focusing on one at a time while maintaining the others has been successful for me.

Understanding Deployment Frequency

Deployment frequency measures how often a team puts code into production. This is distinct from release frequency, which refers to how often users receive new features. In a high-performing team, these are separate processes that happen independently: engineering handles deployments, while product management handles releases.

It’s often said that a failed test is just data. When you’re able to deploy multiple times a day, this is true. However, if you can only deploy on a monthly basis, the cost of being wrong increases. Negative results cease to be data and start to feel like failures. As a result, teams become slower because the need to achieve positive results from each test rises, leading to more time spent planning and less time developing—a vicious cycle. Deploying more often reduces the cost of negative results, alleviating risk and making the team more comfortable with experimentation.

As with DevOps in general, increasing deployment frequency isn’t simple. It requires changes in multiple areas including; tooling, processes, culture, and sometimes architecture.

Best Practices for Improving Deployment Frequency

Implement Feature Flagging: If you don’t have a way to deploy code to production without exposing it to users, the stakes for each deploy are too high. This can lead to larger, riskier deployments and longer lead times. Feature flagging separates deployment from release, allowing you to deploy partial features safely. Once you can deploy behind a flag, you have the freedom to deploy smaller changes more frequently.

Make Stories Smaller: Define smaller units of work for your team. Smaller, independently deployable units of work can be deployed more frequently and are inherently less risky.

Evolve Architecture: Your architecture should support rapid deployment. If multiple teams are working on a single codebase, aim for each team to be able to deploy independently. In a monolithic codebase, teams often experience delays waiting for others to deploy. Breaking up your code into services or microservices that are independently owned and deployed allows for faster deploys.

Push-Button Deploys: The ultimate goal is to achieve CI/CD, but in the meantime, make your deployments as easy and repeatable as possible. Easy deploys reduce friction and limit errors during deployment. The effort to simplify your deploy process will pay dividends when you transition to fully automated deployments.

Automated Testing: Investing in automated testing reduces QA cycle time, allowing for faster deploys. While it may seem obvious, it’s worth emphasizing. You don’t need to target 100% automated testing—each incremental investment in automation will pay off, so continue increasing your coverage. Note that automated tests come with overhead; the more coarse-grained the test, the higher the cost. Develop your test suite with the majority of testing at the lowest levels and reserve end-to-end testing for the happy path.

A Note on Failure Rate

Failure rate is outside the scope of this post but is impacted by deployment frequency. The fear of introducing instability is likely the top concern when increasing deployment frequency and moving towards automated deployment.

As you deploy more often, you might introduce failures more frequently. However, this doesn’t mean your users will experience instability. This concept can be counterintuitive, but as you deploy more frequently, the scope of your deployments are smaller, and the time between a failure’s creation and its discovery is shorter. These factors make it easier to address failures before they broadly impact your users. With good monitoring, you can often address failures before any users are affected.sier to address failures before they broadly impact your users. Add some good monitoring to the mix and you can often address failures before any users are affected.

My Experience Improving Deployment Frequency

In a recent position, my team and I improved our deployment frequency from once a month to multiple times per day. The impact on the company was profound. Monthly deployments meant merging large amounts of unrelated code from multiple teams, which increased the potential for merge conflicts. The testing burden for an entire month of code was immense, and addressing failures was slow. Bugs introduced weeks earlier were difficult to isolate, recall, and fix. The product team, knowing they could only deploy once a month, was hesitant to experiment. They needed to get everything right the first time, leading to overbuilding and over-specifying features.

To begin, we implemented several practices, starting with feature flagging and small stories. We focused heavily on improving deployment frequency first and found that our failure rate also improved simply because we increased the number of deployments without creating more failures. We spent time changing the architecture so that each team could deploy code independently. Eventually, we implemented full CI/CD, enabling us to deploy multiple times per day.

The biggest takeaway from this experience was that focusing on improving one of the four DORA metrics provides clarity and direction for your DevOps journey. Before focusing on deployment frequency, it felt like we were making many improvements that weren’t necessarily getting adopted or having a demonstrable impact. Concentrating on a single metric lit the path for us and made it easier for the team to follow. My second biggest takeaway was that it’s hard for anyone to fully grasp the value of such improvements until they experience it themselves.

Conclusion

Improving deployment frequency is a pivotal step in enhancing your team’s performance and driving your company’s success. By focusing on this key DORA metric, you can foster a culture of continuous improvement, reduce the risk associated with large, infrequent deployments, and make your team more comfortable with experimentation.

Remember, increasing deployment frequency is not just about making technical changes. It requires a holistic approach that includes refining your processes, evolving your architecture, and cultivating a culture that embraces rapid iteration and learning from failures.

From my own experience, the journey to higher deployment frequency can significantly transform your development cycle and deliver tangible benefits quikcly. Start small, focus on incremental improvements, and watch as your team’s efficiency and confidence grow. The path may be challenging, but the rewards are well worth the effort. Embrace the journey, and you won’t regret it.