Process and Scope of An Assessment
The assessment project uses an iterative approach to explore the relationship between incidents, incident responses, post-incident analysis and synthesis, and how the resulting learning is captured and retained across the workplace. The process is case-based and uses incidents as cases. Iteration brings the critical elements to the fore quickly and generates focus.
We typically anticipate four cycles of activity, approximately two weeks per cycle.
|CYCLE 1||CYCLE 2||CYCLE 3||CYCLE 4|
|Initial exploration of existing practices and state of learning from incidents||Deep-dive on 1-2 incidents, tracing their influence on anticipation/prep for future events||Focused exploration based on Cycle 2 findings||Focused exploration based on Cycle 2/3 findings, and synthesize results into artifacts and/or training.
The first iteration is used to trace and map the formal, nominal, and typical after-incident analyses and sharing processes across a group of incidents, focusing on recent events.
During this iteration, we review:
- How incidents are identified, managed, and logged
- How the individuals and groups responsible for management analyze, makes sense of, and classify incidents
- How this information flows into formal and informal incident handling systems
- How further investigation and development of countermeasures takes place
- How countermeasures are prioritized
- How these activities vary depending on severity, complexity, and urgency
The result of this work is a map of incident handling and an incident collection that characterizes the recent company experience.
Cycle 2 will identify opportunities for a deep look at one or two events, and explore how preparation for upcoming events and future designs are formed. Planning for this activity will occur at the end of Cycle 1.
The results of Cycle 2 will be used to plan the following cycles.
Detailed planning for Cycle 3 and 4 occurs at the end of Cycle 2.
Some typical candidate areas for attention are:
- What are the current sources of insight and hypothesis generation during and after incidents?
- How are countermeasures developed and chosen?
- Where is the “memory” of important past events?
- What themes are present in (and absent from!) post-mortem reviews?
- How does anticipating future events shape the work to prepare for them?
- What can an organization do to build capacity around post-incident debriefings (“post-mortems”)?
- What makes an incident a good case for post-incident review?
- What tools are needed to make post-event reviews efficient?
- What influence (positive or negative) do post-incident review have concerning bringing new employees up to speed with current risks, esoteric knowledge of systems behavior, etc.?
- How does authority and/or responsibility migrate in the midst of responding to an incident?
Even though the focus of the project is an assessment, this project itself tends to create momentum for change within an organization:
- Focusing attention on learning from incidents sends a powerful message to the team about the importance of post-incident review.
- Engagement between ACL and engineering staff in dev and ops will generate interest in related topics, new routes of expression, and energy that can be used to move forward in this area.
Ongoing opportunities during the project
During the cycles, there might be opportunities to capture data from new incidents. Our continued presence at the company makes it possible to take advantage of these opportunities to observe the response to incidents, the after-incident reactions, the production of specific countermeasures, and the dissemination of lessons learned from these ‘live’ events.
Naturally, we stay in contact throughout the project with stakeholders. There are usually short progress/midcourse correction meetings near the end of each cycle. These sessions keep the leadership team up to date on the project.