Using data to diagnose and fix broken engineering processes
The following is a guest post from Code Climate
Even the best managers fall prey to the Fallacy of Process — the idea that processes are inherently valuable because they help keep the team coordinated and make workflows more predictable.
In reality, broken processes can be incredibly destructive, creating inefficiencies that hamper your team’s productivity and frustrate developers.
Teams that are able to spot and fix broken processes are more efficient, more productive, and ultimately, happier.
Even though broken engineering processes can be difficult to identify, we believe data can help. Managers can find opportunities for improvement by combining quantitative metrics drawn from your VCS with qualitative information gathered from your team members. In doing so, you can work with your team to improve your workflow and remove bottlenecks for your engineers.
Start with the big picture
If you’re not sure where to start, a mental model can help.
Outline the way work should be flowing through your software development pipeline, and share that model with your team. Ask for their feedback, and pay attention to those instances where their own understanding of the team’s workflow conflict with yours — these misalignments could point to an area of your pipeline that needs attention.
Break down your pipeline
It can also be helpful to break your development pipeline down into phases. With an engineering analytics platform like Code Climate’s Velocity, you can look at some key engineering metrics associated with each phase and determine whether your team’s work is trending in the right direction. If things aren’t running how you’d expect, you’ll want to dig deeper into that part of the process, and look for opportunities to make improvements.
We break the software pipeline into four phases:
Time to Open. This is the time that elapses between a developer’s first commit and when they first open the associated Pull Request (PR). A holdup at this stage could indicate that project specifications are unclear, or that developers need some coaching on good coding hygiene. If developers aren’t pushing commits at all, it could indicate that they’re not getting enough time to work.
Metrics for Time to Open include Commit Volume and PR Size
Time to First Review. The time it takes for an open PR to be picked up for review. Slowdowns here can indicate that developers aren’t prioritizing code review, or that they don’t have time for it.
Review Speed is the key metric for this part of the process
Time to Approve. This is a measure of how long it takes for a PR to make it through Code Review. If your review process is slow, it could indicate a lack of alignment, or a knowledge silo.
Look for holdups in the review process with metrics like Review Cycles or Review Distribution
Time to Deploy. This is the time it takes for code to be deployed, once it makes it through review. Holdups here are generally due issues with technical tooling.
These metrics can be found in your CI tool, and revolve around error rates and build speeds
Talk to your team
Once you’ve identified the parts of your pipeline that need attention, it’s important to discuss them with your team. Quantitative data can point you towards a process that isn’t working, but it can’t tell you how to fix it. Use standups, retros, and 1 on 1s as an opportunity to discuss your findings and brainstorm solutions.
For example, engineering data can tell you that your team’s Review Speed is trending upward, and that individuals are taking much longer to pick up PRs for review, but it can’t tell you why this is happening. However, you can still use that data to serve as a jumping-off point for discussions with your team. You might find that most of the developers on your team are unfamiliar with a particular area of the codebase, and tend to leave related PRs for the one member of the team who is comfortable with that portion of the code. With that information, you’ll understand the source of the issue — a knowledge silo — and have a better sense of how to fix it.
Involving your team in this process has the added benefit of kicking off the Virtuous Circle of Software Delivery. As your team members reap the benefits of process optimizations, they’ll be motivated to look for other opportunities to improve.
Gather your own metrics
Data about your team’s engineering processes and workflow already exists in the place your team does its work — your VCS. Though it can take a bit of work to aggregate and analyze the information, your VCS contains information about everything from your team’s average Pull Request Size, to your Code Review process, to your throughput.
An engineering analytics platform like Climate’s Velocity can make it easier to gather and interpret the data generated by your team’s development pipeline. Sign up for a free trial, and we’ll show you how over 50 metrics drawn directly from data in your team’s VCS and ticketing software can help you understand your team’s workflow and identify opportunities for coaching and improvement.