Remember when we measured lines of code as an indicator of developer productivity? Code coverage? Remember when your developers used to merge a feature branch into main about once or twice a month, with a major change that required just one (albeit large) code review?

Everything changes, and the role of software engineering leaders is no exception. Our team members are using gen AI to produce more code, more quickly. Our development practices have changed—for the better—to get smaller changes into production much more frequently. That all means a different set of data that you need to keep an eye on, and increased concerns about issues like code complexity, security issues, and tech debt.
‍

AI is firmly a part of the software development lifecycle

Within engineering teams, the overwhelming majority of people are using AI in their development work. The 2024 DORA report (one of my favorite reads) shows 75.9% of respondents using AI for some aspect of their work, and 74.9% using it for code writing tasks.¹

This isn't just a trend. It's here to stay. Within Flux, our development team uses code co-pilots, but we also consider them as (to use Rachel's term) "mobs of interns".
‍

Getting worse before it gets better

As a leader, you've likely driven your team's adoption of AI because of the benefits—and there are benefits: increased individual productivity, increased individual job satisfaction, increased flow.²

The issues are that the trust in AI-generated code is low. In the DORA report, 87.9% of people said they trusted generated code at some level, but almost 40% reported little to no trust.³ Again, back to Rachel's post, you have to treat AI-generated code with a consistently critical eye. You can't just use the code directly and trust that it's going to do the right thing in the best way.

What I've heard from my conversations with other engineering leaders is the creeping quality issues, the drift over time. Any one changeset is probably fine (albeit larger than we'd like), but folks I talked to voiced concerns about ensuring adherence to best practices and architectural decisions, socializing changes across the team, and managing increasing tech debt—all due to a higher volume and velocity of changes.

And these aren't just gut feels. While individual productivity is going up, there are measurable negative impacts to software delivery. The data show that in software teams that adopt AI, delivery throughput and stability have decreased.⁴

So how can we surface these issues and stay on top of them? It's all about data.
‍

Data-driven decision making

Just typing those words, my eyes involuntarily rolled a little bit. I've been hearing about "data-driven" cultures and decision-making for most of my career. Lots of talk, way too many meetings, usually not a lot of useful action. Frustrated data teams, for companies big enough to staff one of those. 

But within engineering, we do have a lot of this data, a pretty clear handle of what's useful to our work, and the best teams put it to use to iterate and improve. Whether you're tracking the four key metrics of DORA (change lead time, deployment frequency, change fail percentage, and failed deployment recovery time), defect tracking metrics, or performance-focused observability metrics (e.g. latency, throughput, memory usage), you're likely using quantitative data to drive your team's work.
‍

Getting better

What about data that arises from understanding your code? There's benefit in surfacing issues and anti-patterns which your team can tackle proactively:

• potential improper handling of sensitive data, so you can have a senior engineer investigate and see if it's a real problem or an intentional choice (that probably needs better documentation)
• changes in the tech stack (new third party components or libraries), which might trigger a review by one of your architects or your security team
• increase in code complexity, which indicates an area of code that could lead to future bugs, difficult maintenance, and longer get-up-to-speed time for devs new to the codebase
• increased tech debt, where it is, and how it overlaps with other areas of concern, so you can prioritize this work

These types of issues sneak in over time, and only more so with a higher volume of PRs and larger changesets, all driven by AI. You want your team working at full throughput, but as a leader, you need to stay on top of these types of issues.

Here at Flux, we help surface exactly this kind of data as we analyze and evaluate your code, so you and your team can take action on what will have the best impact. Keep using AI to write code, and embrace that the faster your code changes, the more it will be in flux.

‍

References

1. "2024 DORA Report", p.19-20
2. p.32
3. p.23
4. p.39