John Allspaw

A note about terms: Use of terms like anomaly, event, incident, and accident tend to evoke strident debates about their exact meanings. They are used inconsistently in tech and elsewhere. Frustration with their variable interpretation has led some to try to give them crisp definitions.

Despite these efforts, none of these terms have fixed meanings. The situation is made even more difficult when word choice has significant consequences. For example, some tech firms have formal processes for handling an incident that do not apply to an event or an anomaly.  

(We have witnessed extensive discussions during event response about whether that event meets the organization’s threshold criteria for an incident! Declaring an incident would bring additional resources to bear, generate auditable documentary trails, and involve substantial future work.)

The situation cannot be resolved by fiat. Instead, we need to pay attention to how these terms are used in context and especially to the consequences of the choice of term.

In this chapter, we use the term incident as a pointer to a set of activities, bounded in time, that are related to an undesirable system behavior.

This is a keynote talk I gave at the first Learning From Incidents conference in 2023. The video recording is here. It seems clear to me that industries and organizations that depend critically and fundamentally on software have been paying attention to the sort of topics people have spoken about here at this conference. The

The following is a transcript (edited just a tiny bit) and slides from a talk I gave at SRECon 2024 Americas. For those who like videos, here it is. Before we get started, could you please raise your hand if you’ve read and are familiar with a paper entitled “How Complex Systems Fail”? If you’re

Multiple professional and research communities feel a profound loss at the death of Richard I. Cook. Richard died peacefully at home on August 31, 2022 in the loving care of his wife Karen and his family. Dr. Richard Cook was a polymath who excelled in multiple careers, usually simultaneously. A physician and anesthesiologist, he was

Dr. Johan Bergström, who leads the MSc program in Human Factors and Systems Safety at Lund University (I am an alumnus) has a short ~7 minute video discussing three common analytical traps that incident analysts and accident investigators can get caught in. They are: 1. Counterfactual reasoning2. Normative language3. Mechanistic reasoning Have you seen any of these

These are just a few “Above-The-Line” patterns we’ve observed in cases over time, and potential nicknames for the patterns in the style of how cons are named in movies such as Ocean’s Eleven. Is this shorthand that we use in regular use in our work? Not regularly, but it’s been fun to name them once

I am an angel investor in Jeli.io, and could not be more excited for the product to come out of “stealth” mode, for many reasons! At its core, Jeli is built around the easy collection and annotation of this data, helping analysts make connections between chat transcripts, interviews, prior related incidents, and a whole host of

A few weeks ago I gave a talk at the DevOps Enterprise Summit London (which was virtual). The description of the talk: In the past two years, we have had the opportunity to observe and explore the real nitty-gritty of how organizations handle, perceive, value, and treat the incidents that they experience. While the size,

I was honored to present a talk at the AllTheTalks conference a few weeks back. tl;dr: slides are here, video is below The topic was incident analysis (big surprise there!) and the notion of learning and fixing, and how these activities are related but not the same. A key idea here is that rather than focusing on simply focusing on identifying fixes

I was invited to give a talk at the Spotify office in New York last month on the topic of learning from incidents. Here is the first slide, which I hope to be the BLUF (“bottom line, up front”) of the talk… Here is the video recording of the talk…