It's Not About the Code: 5 Surprising Truths AI Reveals About Engineering Teams
Stop measuring AI by utilization. Learn why AI is actually a diagnostic tool that exposes legacy code issues, forces better DevEx, and shifts engineering bottlenecks.
Stop measuring AI by utilization. Learn why AI is actually a diagnostic tool that exposes legacy code issues, forces better DevEx, and shifts engineering bottlenecks.
The prevailing narrative around AI in software engineering is one of hyper-productivity. We hear about AI copilots writing code faster, debugging more efficiently, and accelerating development cycles. While these advancements are significant, they only scratch the surface of AI's true impact. The most profound changes aren't happening in the code editor, but in the executive suite, the team meeting, and the project planning process.
AI's integration into engineering workflows is acting as a powerful diagnostic tool, revealing deep, often uncomfortable, truths about how we manage, measure, and structure our teams. It's forcing a re-evaluation of long-held assumptions and exposing the hidden inefficiencies that have plagued software development for years.
In a conversation between Justin Reock, Deputy CTO at DX and Alireza Boloorchi, Founder & CEO of Catch Up AI, one theme kept repeating: AI is less of a generator and more of a diagnostic tool.
Think of AI as a high-contrast dye injected into the bloodstream of your engineering organization. It doesn't just speed things up; it highlights every blockage, every broken process, and every measurement failure you've been ignoring for years.
Here are the 5 surprising truths AI is forcing engineering leaders to finally confront.
1. The "Expensive" Irony: Why AI is Finally Forcing Companies to Value People
For years, advocates for a better developer experience (DevEx) have argued for the importance of good documentation, low code complexity, and well-organized repositories. Yet, these initiatives were often seen as "nice-to-haves" and perpetually underfunded. As Alireza Boloorchi notes, executives historically viewed DevEx as a "lagging indicator," making it a low priority when immediate results were demanded. The core irony of the current AI boom is that the massive financial investment it demands is forcing executives to finally care about these foundational practices.
With companies pouring huge sums into AI tools, leadership is demanding a measurable return on investment (ROI). This scrutiny has inadvertently placed a spotlight on DevEx. As Justin Reock explains, the very same conditions that create a good experience for a human developer, clear documentation, clean architecture, low complexity; also create a good "agent experience." AI tools are more accurate, efficient, and effective when they operate in a well-maintained environment.
It took the hype and expense of AI to push companies to invest in the foundational engineering practices they should have been prioritizing for their human developers all along. This expensive technological wave is forcing a return to human-centric fundamentals.
It means good things for the average developer and for developer experience... just maybe a little bit of a shame that this is what it took to finally get us to care so much about this.
The Takeaway: The high cost of AI is inadvertently funding the human-centric improvements engineers have wanted all along.
2. The "Utilization Trap": Why Your Metrics are Broken
How do you measure the success of an AI rollout? If your answer is "Utilization Rate" (i.e., how many devs are using it), you are walking into a trap.
Boloorchi recalls the panic of the pandemic shift to remote work, where leaders, blinded by a lack of visibility, resorted to counting "commits" to measure productivity. We are seeing a repeat of this mistake. If you grade engineers on whether they use AI, they will use it; even when it's not helpful, just to show up on the dashboard.
A Better Framework for Measuring AI To avoid "Shadow AI" (where devs use tools secretly) or "Fake Usage" (where devs use tools uselessly), you need a 3D measurement framework:
- Utilization (Baseline): Who is using it? (Just to know adoption).
- Impact (Velocity/Quality): Is Pull Request (PR) throughput increasing? Are revert rates dropping? Is the needle actually moving?
- Cost (ROI): Do the gains in step 2 justify the bill in step 3?
3. The "Bottleneck Shift": Fixing Code Reveals the Real Problem
There is a concept in systems engineering called the Theory of Constraints: Every system has one primary bottleneck. If you fix it, the bottleneck simply moves to the next slowest step.
AI has dramatically uncorked the "Code Generation" bottle. But as Reock points out, "We fixed a bottleneck... what's the new bottleneck?"
If your developers can now move 15x faster, can your Product Managers write requirements 15x faster? Can your QA team test 15x faster? The answer is usually no.
The bottleneck is shifting upstream to Product Ideation and Requirements Gathering. The problem isn't writing the code anymore; it's feeding the "Developer Factory" with enough clear, high-quality specs to keep the machines running. AI is exposing that our planning processes were slow all along, we just didn't notice because the coding took so long.
4. AI Creates More Engineering Jobs, It Doesn't Destroy Them
Contrary to the prevailing narrative of replacement, the economic principle of "induced demand" offers a more likely scenario for the future of engineering roles. As Justin Reock explains, history shows that when a technological breakthrough makes a process more efficient, it doesn't lead to fewer jobs. Instead, it expands the realm of what is considered possible, creating new ambitions and, consequently, more work.
As AI dramatically increases the capacity of what a single engineer can accomplish, organizations will not simply do the same work with fewer people. They will tackle more complex problems, build more ambitious products, and pursue goals that were previously out of reach. This expansion of scope will ultimately increase the overall demand for engineers needed to manage, architect, and oversee these larger and more complex systems.
The result is not a reduction in the engineering workforce, but an elevation of its role and an increase in its numbers.
We induce more demand for engineers because we can do more with the software than we could do before and so we end up creating more engineering jobs than... reducing.
5. Your Legacy Code is AI's Kryptonite
Demos of AI agents are seductive because they almost always show "Greenfield" projects, building a new app from scratch.
The Reality Check: Most engineering happens in "Brownfield" environments, decades-old codebases, tangled dependencies, and abstract layers that, as Reock notes, were designed for human cognition, not machine parsing.
This legacy code is AI Kryptonite.
Large Language Models (LLMs) struggle to maintain context across millions of lines of fragmented, legacy infrastructure. This forces developers to slow the AI down, using "patch-based prompting" rather than letting the AI run wild.
The Market Split: This creates a fierce battle. Incumbents have the data and customers but are slowed down by legacy code. Startups are "AI-Native" and fast, but lack the distribution.
Conclusion
AI is proving to be a mirror, not just a motor.
Its true value lies in the difficult conversations it forces us to have. It is compelling us to fix our measurement systems, invest in developer experience, and identify where our product pipelines are truly broken.
The ultimate question for engineering leaders is no longer "How can we code faster?" We solved that. The question is now: "Since we can build almost anything, have we gotten any better at deciding what is actually worth building?"
