Leadership at the Edge

Introduction

In the fast-evolving world of technology, the drive to deliver quickly and efficiently is more pressing than ever. However, the quest for speed often comes with its own set of challenges, particularly when it means compromising on quality. This is where DevOps Research and Assessment (DORA) metrics come into play, offering a balanced framework that measures both the pace and the health of software development and operational performance.

DORA metrics consist of four key indicators: Deployment Frequency (DF), which gauges how often an organization releases to production; Lead Time for Changes (LT), the time it takes for a commit to go live; Change Failure Rate (CF), the percentage of deployments that fail in production; and Mean Time to Recover (MTTR), how quickly a team can recover from a failure. These metrics are crucial for optimizing development processes and driving high performance without sacrificing quality.

My journey with DORA metrics began at a dinner hosted by a DevOps metrics company, during a period when I was struggling to establish a system of metrics that promoted delivery performance without fostering negative incentives or glossing over systemic issues. The concept of “metrics and counter metrics” was a revelation. The philosophy is straightforward: if you aim to enhance a particular aspect of your team’s performance, you should also track the metric that could deteriorate if the team overly focuses on that single aspect. For instance, while speed is essential, it shouldn’t come at the expense of quality.

DORA addresses this by balancing two speed metrics (DF and LT) with two quality metrics (CF and MTTR). This balance ensures that if a team reduces lead time at the cost of increasing the change failure rate, it doesn’t count as an improvement. True progress is achieved when teams enhance their speed while maintaining or even improving quality, embodying a holistic approach to performance improvement.

The Four Key DORA Metrics

• Deployment Frequency (DF): Deployment frequency measures how frequently the team puts code into production. Good teams do this between once a week and once a day. Great teams deploy multiple times per day.
• Lead Time for Changes (LT): Lead time measures how long it takes from an engineers first commit on a task until it is in production. Good teams are between 1 week and 1 day. Great teams average less than 1 day.
• Change Failure Rate (FR): Change failure rate measures the percentage of deploys that lead to failure. A failure is defined by the business, but it is usually related to support escalations and or monitors being triggered. Good teams are between 15%-30%. Great teams are less than 15%.
• Mean Time To Recover (MTTR): Mean time to recover measures how long it takes to relieve the failure. This doesn’t necessarily mean fixing the underlying issue, just allowing work to continue. Good teams restore service in less than 1 day. Great teams restore service in less than 1 hour.

Implementing DORA Metrics in Your Teams

Implementing DORA metrics within your organization can present a significant challenge, especially when it comes to aligning the metrics with your current development practices. From my experience, while some teams choose to develop their own tooling, using a third-party product that automatically tracks these metrics can often lead to more reliable and less biased results.

I’ve found the most success with third-party tools that are somewhat opinionated about how metrics should be tracked. Initially, these tools might reveal less flattering metrics, but as your organizational practices mature, so too will your metrics. For example, in environments where teams do branch-based development and batched deploys, it can be complex for any tool to accurately assign credit for deployments or blame for failures. Often, all involved teams are credited or blamed, which, although frustrating, underscores the need for moving towards more agile practices like CI/CD and trunk-based development. The key is not to tailor your tooling or metrics too finely to fit outdated processes but to allow them to guide and reflect your progress towards more efficient practices.

The failure rate can vary significantly from one company to another, largely because the definition of failure is uniquely tailored to each company’s operations. Initially, the simplest approach is to track failed builds. As your metrics tracking and practices evolve and mature, you can expand this to include support escalations and, eventually, alerts from your observability tools. This progressive enhancement of metrics ensures that your definition of failure evolves in step with your operational improvements.

Key Tools and Practices for Implementing DORA Metrics

CI/CD Pipelines: Continuous Integration and Continuous Delivery pipelines are crucial for improving both Deployment Frequency and Lead Time for Changes. They automate the build, test, and deployment processes, ensuring that new code changes are automatically prepared and shipped to production, thus speeding up the entire cycle.

Automated Testing: High-quality automated tests are essential for maintaining a low Change Failure Rate. They help catch issues early, reduce the risk of defects reaching production, and increase the confidence of the team in the codebase.

Monitoring Tools: Robust monitoring and alerting systems are indispensable for reducing the Time to Restore Service. These tools enable real-time observation of the production environment and can quickly alert teams to issues, speeding up incident response times.

Version Control Practices: Adopting trunk-based development practices where developers merge their changes into a shared trunk regularly, multiple times a day, can significantly enhance the visibility and traceability of changes, supporting better metrics across the board.

Embracing Feature Flagging as a Key Improvement

One of the first and most critical improvements to implement following the setup of DORA metrics is feature flagging. Feature flagging allows for a clear separation between deploying code and releasing features to users. Traditionally, when code is deployed, it is also released, directly impacting the user experience. This can be disruptive, especially if your deployment strategy targets multiple deploys per day.

With feature flags, you can deploy at will, while leaving the decision on when and how to release features to product and marketing teams. This approach not only accelerates deployment frequency but also allows engineers to deploy incomplete features safely, facilitating early user access for testing. Moreover, feature flags greatly enhance your Mean Time to Recover (MTTR) as they provide a quick way to mitigate issues by disabling the problematic feature until a fix is implemented.

Although integrating feature flagging requires a shift in development practices, the trade-off of greater agility and control over software releases—is immensely beneficial.

Lessons Learned from High-Performing Teams

While tracking metrics and implementing necessary DevOps interventions are crucial, they alone do not guarantee the formation of a high-performing team. The true essence of a successful team lies not just in the metrics, but in the people behind these metrics. Here are some vital lessons learned from leading high-performing teams through significant technological and cultural shifts:

Commitment to Continuous Improvement

A high-performing team is characterized by its commitment to continuous improvement and the resilience to endure the challenges that come with it. This commitment requires more than just technical skills; it necessitates a mindset geared towards growth and adaptability. It’s essential that each team member not only understands but also buys into the long-term vision of the organization’s DevOps journey.

Belief in the Vision

The allure of methodologies like CI/CD is undeniable, promising faster deployments and more robust systems. However, these benefits come with substantial challenges. For a team to fully engage with such a transformative process, there must be a profound belief in the vision. Without this foundational belief, progress can be sluggish, and initiatives may falter. As a leader, it is your responsibility to continually articulate and reinforce this vision, making clear the tangible benefits not just to the organization but to the team members themselves.

Highlighting Progress and Value

Regularly highlighting the progress being made is vital for maintaining morale and motivation. It’s important to connect the dots between the team’s efforts and the value these efforts add to the company. Recognizing achievements, no matter how small, can reinforce the worth of the hard work being done. This not only fosters a sense of accomplishment but also aligns the team’s daily activities with the broader goals of the organization.

Understanding the Journey

Perhaps one of the most crucial understandings for any high-performing team is that the journey towards DevOps excellence is ongoing—there is no final destination. The objective isn’t to architect and build a perfect system but to continually evolve and improve upon existing processes. This perspective helps prevent complacency and keeps the team focused on leveraging improvements for competitive advantage.

Encouraging Continuous Improvement

Cultivating an environment that embraces continuous improvement is essential for any organization aiming to stay competitive and innovative. This culture not only fosters resilience but also encourages a forward-thinking mindset among team members. Here’s how leaders can actively promote and sustain this culture:

Keeping Score and Connecting the Dots

One of the inherent challenges with continuous improvement is that individual efforts might seem insignificant on their own. It’s crucial for leadership to not only track these improvements but also to regularly illustrate their cumulative impact. By demonstrating how each small change contributes to significant enhancements over time, you help the team visualize their progress and understand the broader impact of their efforts. This visibility is key to motivating the team and embedding a mindset of ongoing development.

Encouraging Experimentation and Accepting Failure

Innovation inherently involves risks and the possibility of failure. To truly foster a culture of experimentation, it’s vital to create an environment where team members feel safe and supported in taking these risks. Encourage your team to experiment with new ideas and approaches, reminding them that failure is often a stepping stone to greater discoveries. Make it clear that it’s acceptable to fail and that each failure provides valuable lessons.

Celebrating Failures and Learning

When failures occur, it’s important to celebrate them, focusing on the learnings and the efforts made. Highlight how each failed attempt provides insights that can lead to future successes. Ensure that team members know they are supported not just in word but in action, creating a trust-based environment where taking calculated risks is encouraged.

Building a Feedback-Rich Environment

Regular feedback is a cornerstone of continuous improvement. Encourage open dialogue and regular feedback sessions where team members can share their thoughts on what’s working and what isn’t. This feedback should be constructive and aimed at processes and outcomes, not individuals and should be supported by the metrics. Such an environment not only helps in adjusting strategies quickly but also ensures that everyone feels their voice is heard and valued.