Here’s the gist of this post:
- AI adoption is outpacing the industry’s understanding of how it makes human cognitive work more critical, not less.
- AI technologies are often assumed to take over work people do, keeping the current system as-is, while improving results. This is known as the substitution myth, and reality (and research) shows it to be false.
- People are now required to step in and rectify situations the AI cannot handle, at exactly the moment when they are least prepared to do so.
If you hadn’t noticed, many every company on the planet is in the process of adopting generative AI. The main rationale for this is because it promises to automate work, reduce headcount, and other costs by taking over activities that have historically required human expertise. By all accounts, AI adoption is accelerating.
The expectations about the value of this adoption are rising just as fast, especially among investors, leaders, and other stakeholders who expect the use of AI will reduce headcount and related costs.
Interestingly, the result of this explosive growth of usage and expectations is that human cognitive work becomes more critical, not less.
You might be thinking “Wait, what?” Indeed, this runs counter to what most people expect.
The common belief, known as the substitution myth:
The substitution myth is the assumption that a technological function (such as an AI agent) can replace a human function without fundamentally changing the work system in which that function is performed.
It is a myth because in reality, it plays out a bit differently. Instead of a one-for-one swap (where people doing the work are swapped with an AI agent doing the work), a number of new and different requirements emerge.
Ok, I am going to pause here for a moment. My expectation is that most people reading this are already familiar with the substitution myth. Despite this, I suspect that more readers than I expect are hearing about it for the first time. In any case, I would consider this to be among the most fundamental concepts to understand in this period of explosive AI growth and attention. Now, back to the post…
First: people have new tasks that didn’t exist before.
Obviously, in order to get any value from AI technology, there are a bunch of things that need to happen to get it all going. Someone has to do things like:
- configuring the new systems: agents, models, harnesses, parameters, etc.
- integrate it with existing systems
- monitor whether it’s working correctly
- etc.
In the case of software development, there are a number of new things engineers need to learn and do that they did not have on their plate before introducing AI.
Software engineers who have worked with LLMs for only a short time will likely recognize many if not all of these new skills to learn and new tasks to take on. These include things like:
- Learning how to write and fine-tune prompts in specific ways to guide the LLM’s code generation. This tends to happen iteratively, over a period of time, before someone can be genuinely effective at structuring requests and knowing how to refine them so they produce good results.
- Reviewing and evaluating the agent’s responses is also not intuitive and can differ across different models, harnesses, etc. By this I mean something much broader than just reviewing generated code. Recognizing when an agent will happily continue working on a task when it’s not warranted, detecting when a confident response about an uncertain topic is too sure of itself, etc.
- Evaluating, choosing, and confirming which models, agents, tooling, etc. is appropriate for which type of work.
- Detecting hallucinations, including those that do not affect the functionality at the moment, but almost certainly will confuse anyone else who may need to work in this part of the codebase in the future.
- (I’m sure there are many other examples of new skills and new work)
I have noticed that many engineers don’t really acknowledge these things as new skills or new tasks. I suspect that it might be for a few reasons, but that’s for a different post. Regardless, these represent very real investments of time and attention.
Okay, if the assumption was that replacing people with AI would “free them up” for other tasks, it should seem clear that this is not the case.
Second: people’s roles change.
People shift from being authors to reviewers, from operators to supervisors. People are no longer operating their system, they are now monitoring a new system which operates their system.
Shifting from creating content (code, documentation, diagrams, etc.) to verifying generated output is so significantly different that multiple studies have looked closely at the mental fatigue, cognitive strain, and many other negative effects on engineers’ well-being[1].
This of course is profoundly different (and more difficult) work than they had before.
But this is not the most challenging implication for people…
Third: the context in which people’s work has shifted.
Over time, people who previously performed the tasks now handled by agents become less familiar with the details, so their grasp of how the work is done begins to fade. This shift is not specific to AI, it’s just what happens to skill when we’re out of regular practice with them. What is specific to AI is that the key details become mediated. In other words, people are able to know how the work is done only indirectly, through the agent.
As a result of all the above, when things do go sideways and the AI agent isn’t able to cope with the situation, people will need to step in and rectify the situation.
This at exactly the moment when they are least prepared to handle it.
They are now in the worst position to deal with surprising situations (like incidents) because they’re no longer familiar with how the stuff works.
We’re asking people to handle AI-era incident situations with pre-AI-era tooling
Because of the persistent belief in the substitution myth, many observability, telemetry, and other SRE-focused products have focused on developing and integrating what have been called “SRE AI” agents as the primary (if not exclusive) product direction for the future. When basically all product development attention is spent on this, it’s tough to find space and time to work on much else, never mind projects aimed at supporting people.
The architectures, designs and functionality of observability, monitoring, and incident handling tooling can’t remain the same if the people using them need to step in when AI cannot cope with a disruption. They need to be adapted in particular ways in order to effectively support the people responsible for dealing with surprises when the AI cannot.
Traditional product development approaches will not advance pre-AI tools enough to support people in the future. It will require different approaches and perspectives than traditional product UI and UX design to develop the necessary functionality and adapt interfaces to support this more challenging and consequential cognitive work.
Ok. To recap what we’ve covered so far:
- AI adoption is accelerating, and the expectations of a glorious future accompanies it.
- A surprising twist appears: putting AI agents in place and using it effectively takes real effort and creates new burdens.
- When things go well, gaining new capabilities comes with being less equipped to deal when things…don’t go well.
- The existing and well-worn approaches to product development will not help us.
What if I told you there was a field born to provide the methods and perspectives needed for this AI era?
Cognitive Systems Engineering (CSE) is a field that emerged in order to meet the critical challenge of supporting people working in complex, consequential, and fast-moving environments. It produced completely new concepts, methods, and techniques required by people, working jointly with non-human agents, in domains where automation only becomes more opaque, inscrutable, and dangerously capable over time.
And yet…it is a field almost entirely unknown in mainstream online software engineering. Most in the industry have never heard of CSE, and the relative few who have tend to dismiss it as simply “good product design.”
I mentioned to my colleague David Woods recently that I can make a solid argument now that while CSE was born from what the Three-Mile Island accident demonstrated to the world…maybe it’s been working since then to be ready for the moment in time we are in now.
There is more to come on many of these topics above, but I will end here with an observation I’ve made many time since ACL first started. While CSE perspectives are extremely powerful, they are also so incredibly different from what software engineers are familiar with that it often takes effort to understand them
However, once it does become clear, they cannot see the world in the same way they did before.
1. This is only a short list of the fast-growing research on these topics:
Baltes, S., Cheong, M., & Christoph, T. (2026). “An endless stream of AI slop”: How developers discuss the burden of AI-Assisted software development (2603.27249V4). arXiv. https://arxiv.org/abs/2603.27249v4
Fairuz Nawer Meem, & Johnson, B. (2025). Investigating the impact of AI-Assisted tools on software practitioner well-being. CHIWORK ’25 Adjunct: Adjunct Proceedings of the 4th Annual Symposium on Human-Computer Interaction for Work, 1–5. https://doi.org/10.1145/3707640.3731915
Tripp, J. (2026). The supervisory paradox: How AI delegation transforms productive labor into cognitive vigilance. https://doi.org/10.2139/ssrn.6503239
Vahid, G. (2026). Human oversight and overload: Two hidden and costly burdens of AI-Assisted software engineering (2606.05770V1). arXiv. https://arxiv.org/abs/2606.05770v1
