Common reverberations of introducing AI

In the last post, I described the remarkably persistent oversimplification known as the substitution myth: the (usually implicit) assumptions people make about the value AI will bring, often before it’s even implemented.

This post is about what actually happens once the technological change is in place.

In “Anticipating the effects of technological change: a new era of dynamics for human factors,” David Woods and Sidney Dekker describe a sequence they call a pattern of reverberations. It commonly follows the introduction of advanced but opaque (“black box”) technology into complex, critical work environments.

Many people working closely with generative AI these days may find this pattern familiar.

At a high level, it unfolds like this:

  1. The new technology provides new capabilities.
  2. These capabilities raise demands and expectations, so the technology gets used. It also brings new complexities with it.
  3. Practitioners adapt or develop workarounds to cope with these complexities so they can meet the increased expectations others now have.
  4. These complexities and adaptations are surprising side effects. The original design didn’t anticipate or account for them.
  5. Eventually, an unanticipated situation arises that people can’t handle or catch. Their adaptations and workarounds are often incomplete and brittle, partly because of the increased demands.
  6. Because these adaptations and workarounds hide the complexities from others, the system appears (from a distance) to be running smoothly. So when others look back in the wake of an unexpected disruption, they label the failure “human error.”

Let’s see how the pattern might look in situations software engineers commonly find themselves in today. I wanted to see if I could construct some plausible ways that this pattern plays out, in modern day uses of generative AI, so here we go…

In the case of AI-assisted code generation

  1. Developers can now generate boilerplate, tests, and even whole functions from short prompts. This is a genuinely new capability.
  2. The generated code introduces complexity that didn’t exist before: subtly misused APIs, plausible-looking but wrong error handling, the occasional hallucinated package, security anti-patterns that look idiomatic, and so on. Most importantly, the developer is now a reviewer rather than an author. Reviewing code you didn’t write, with no visible (or credible) reasoning trail, is cognitively harder than reviewing code you wrote yourself.
  3. So developers adapt. They develop prompt patterns that produce better output for their particular codebase and situations. They build up their own tricks and rules of thumb, such as:

    ● “Never trust the agent to make changes to auth” or “Always rewrite the error handling”
    ● They also add personal rules for agents to their system prompts, such as: “Check every import actually exists” or “Do NOT, under any circumstances, run git push on f%$@!ing anything”

    Before long, they learn to read agent-written code with a very specific kind of suspicion.
  4. These adaptations are mostly informal, unresourced, and therefore effectively invisible to the organization. (You’re unlikely to find the quirks of engineers’ personal system prompts on the company wiki.) Meanwhile, a second-order adaptation sets in: as the volume of code goes up, the depth of review can go down. Developers may start “scanning and tweaking” rather than genuinely reviewing and evaluating the code, because by now others’ expectations about throughput have absorbed any productivity gains.
  5. Eventually, a vulnerability slips through in a change. It might be a race condition that passes testing but fails under production load. Or a dependency gets added that is unmaintained or, in the case of typosquatting, actively malicious.

    The developers’ adaptations weren’t wrong. They were brittle and incomplete, just as the pattern predicts.
  6. The conclusion, usually drawn by people at a distance who aren’t responsible for this hands-on work, is:

“The developer should have caught that in code review.”

Notice what isn’t visible here. First, there’s the extra burden that lands on developers’ shoulders when the volume of code review jumps significantly. Second, developers’ informal rules of thumb, tricks, and customized workarounds are often the only things keeping the system working. Without them, the failure rate would be much higher.

In the case of AI-assisted incident response

  1. On-call engineers can ask an AI agent to summarize alerts, correlate signals, suggest diagnoses, and draft communications.
  2. New complexities emerge. The agent confidently proposes diagnoses that are wrong in hard-to-notice ways. It summarizes logs but leaves out the anomalous detail a veteran engineer would spot right away. (This is the absence blindness problem; more on that in a future post.) It produces authoritative-sounding, runbook-style text that doesn’t come from any actual knowledge of the system’s expected or current state.
  3. Experienced engineers adapt. They stop expecting the agent to be a crystal ball and start using it more as a brainstorming tool. They cross-check its results. They recalibrate their expectations: “Useful for summarizing a timeline, less so for discovering triggers or causal mechanisms.”
  4. These adaptations depend on deep system knowledge that lets engineers quickly evaluate the agent’s output. Less experienced engineers, who never built that knowledge because the agent was available from day one, treat its responses as their primary diagnostic approach. The agent becomes a de facto proxy between the engineer and the actual system state: an intermediary that nobody designed or evaluated for that role.
  5. One day, an incident escalates because a critical signal was left out of a summary. A wrong remediation is applied because the agent’s confident framing anchored the engineer on a diagnosis that matched a common issue but masked an uncommon one. Things get worse.
  6. The company introduced a tool to resolve incidents faster, rewarded faster resolution, and didn’t recognize (or resource) “verify the AI’s output” as actual work. The engineer’s workaround, which was maintaining an independent understanding of the system while also using the tool, was an unfunded mandate from the start.

    So what becomes the official story about the incident?

“The engineer should have verified before acting.”

Finally, how might the pattern play out when AI is used for a different purpose during incidents?

In the case of LLM-drafted customer stakeholder comms

  1. With very little effort, status page updates, incident summaries, and customer emails can be drafted in seconds.
  2. The output is fluent and confident, but it may or may not be accurate. It might commit to “root causes” that haven’t been confirmed, promise unrealistic timelines, or subtly mischaracterize the scope of the impact. These errors are easy to miss but have real consequences for readers.
  3. So the people who do this work adapt. They develop specific editing routines and build rules to catch certain phrasings. They keep a mental checklist of things the LLM gets wrong.
  4. After a run of AI-drafted communications with no major issues, their trust in the AI and in their own bespoke rules grows to the point where their editing gets lighter. Their checklists get shorter. The task starts to look like a “solved problem.”
  5. One day, a customer communication contains an inaccurate commitment. Or an internal summary misses a detail that changes how leadership understands the situation, which then shapes downstream decisions in less-than-ideal ways.
  6. The takeaway from this embarrassing, messy situation becomes:

“The communications people should have proofread the comms more carefully!”

I wanted to see if I could construct some plausible ways that this pattern plays out, in modern day uses of generative AI. The pattern has that quality where once you see it, you’ll recognize it in places you didn’t expect.

Ok, so how can this help us get something done?

Like all Cognitive Systems Engineering findings, they are not just “good to think with.” There are practical implications the pattern opens up, and Woods & Dekker lay them out for us:

Expect the work to change.

This is simply the substitution myth “in the wild.” The idea here isn’t to not use the technology, obviously. Just don’t create and hold any expectations that implementing it to handle tasks that a person once was responsible for will play out in a simple swapping of AI for the person. It simply does not work that way.

Stop designing for agent autonomy. Instead, steer towards genuinely joint cooperative human-agent systems.

Systems designed deliberately to support human expertise have significant advantages over run-of-the-mill attempts to “free up” the person. The most effective teams and the teamwork they demonstrate aren’t just individuals delegating to each other. They support the work the whole team is engaged with. They are interdependent. See the most successful bands or sports teams across history. Delegation isn’t joint activity. Effective (and efficient!) teams produce results that are better than the sum of the results of each individual.

Recognize that design concepts represent hypotheses, not promises.

Treat your implementations of AI as experiments, in the same way we’ve done continuous deployment for a while now. Do the work to find both disconfirming and confirming evidence for your hypotheses about how things will go, post-deployment. Think about the rationale for running chaos experiments, and do that. If you can build skill in anticipating unintended side effects, you’ll spend time and attention in the short term but will benefit in the long term.

Don’t bank all of your gains from new capabilities as throughput.

Preserve some portion of what you gain and protect that capacity from eroding. Think of it as slack, safety margin, a “slush-fund” of workload relief capacity for when you need it, etc. Regardless of what you label it or what you do with this extra space/capacity, you’ll be happy you did. If you immediately eat the time/energy/attention gains you just received, you’ve made it that much easier for your competitors.

There are more implications that Woods & Dekker outline that are well-worth looking at. But I’d like you to consider that what we’re experiencing, post-deployment, are not unavoidable downsides.

How to maneuver around the dynamics in this pattern has been figured out.
There are answers. There are methods. There are solutions. Just because they’re unfamiliar to mainstream software engineering doesn’t mean you can’t become familiar.

It just requires being a little more humble in how we engineer. We can do that. Right?

Woods, D., & Dekker, S. (2000). Anticipating the effects of technological change: A new era of dynamics for human factors. Theoretical Issues in Ergonomics Science, 1(3), 272–282. https://doi.org/10.1080/14639220110037452

Scroll to Top