Reports: A brief overview

By Gordon Rugg

This article is about a range of techniques which involve people reporting on actions, thoughts or rationale. It’s not about writing technical reports, etc.

These techniques evolved independently, rather than as a systematic toolbox. However, in practice they complement each other neatly. The table below shows how they fit together.

Time Technique(s) Main knowledge issues involved
Future Scenarios Knowing the unknowable
Present Think-aloud Short Term Memory/working memory
Past
  • Critical Incident Technique
  • Hard Case Technique
Active and passive biases in human memory

This article describes how these techniques can be used systematically to identify and clarify clients’ requirements, with particular reference to legal and practical constraints.

All of these techniques involve the respondent reporting on something, as described below. All of these techniques can be used both directly (where the respondent reports on their own actions) and projectively (where the respondent reports on someone else’s actions). The projective approach is particularly useful for exploring sensitive topics.

In scenarios, the respondent is given a description of a situation, and is then asked to say what they would do in that situation (direct report), or what someone else would do in that situation (projective report).

In think-alouds, the respondent is asked either to provide a running commentary on their own actions while carrying out a task (direct report), or to provide a running commentary on someone else’s actions while that other person is carrying out a task (projective report).

In Critical Incident Technique and in Hard Case Technique, the respondent is asked to report on either a critical incident or a hard case from the past, usually with particular reference to what was especially important about the incident or case. Again, this can be done either as a direct or a projective report.

Each of these techniques has its own strengths and weaknesses, described below.

Scenarios

Scenarios are useful for exploring what-if, hypothetical cases. This is particularly helpful if you are dealing with rare events and/or dangerous events, such as a life-threatening problem.

Scenarios can be used systematically in conjunction with other techniques, such as fault or error taxonomies, to investigate methodically what the clients’ requirements are for a wide range of situations. Neil Maiden and his colleagues used this approach for an elegantly systematic analysis of how ATMs (hole in the wall cash points) should handle the various problems that could arise during use.

Scenarios are useful for identifying possibilities. However, the results need to be treated with caution, because people’s beliefs about their own future actions, and the future actions of others, are liable to a range of biases and errors.

Think-aloud

Think-alouds are useful for finding out what’s in the respondent’s Short Term Memory (also known as working memory, though strictly speaking the two aren’t completely synonymous) while performing a task. This is an important advantage, because Short Term Memory (STM) only has a capacity of a few items, and a duration of a few seconds. There’s no chance of getting at the contents of STM via an interview after an event. This is a big issue in fields such as designing software interfaces for aircraft pilots, since the designers need to know what information the pilot will need in any given situation. That’s an STM issue, and the only way of getting at that information directly is by having the person think aloud at the time.

We often combine think-aloud technique with mockups of designs, so that we can get detailed feedback about respondents’ reactions to the designs at an early stage, and identify and fix problems early in the development process. The same approach can also be used to identify problems with existing products and processes, as a first step towards identifying solutions.

Although think-aloud technique is invaluable for accessing the contents of STM, it has various limitations. One is that the output from think-aloud often appears fragmentary and incoherent out of context (for instance, when the respondent is referring to “this part” and “that part” – you often need to video record rather than just audio record, so you can capture the context, but that raises the problem of video recording being more intrusive than audio recording). Another limitation is that respondents usually stop talking when they need to concentrate on a particularly important task, which is precisely where it would be most useful to know what they are thinking. However, if the task in question is something such as landing an airliner, it’s not advisable to insist that the respondent should keep talking; instead, it makes more sense to use a projective version of think-aloud (e.g. asking a pilot to comment on a video of another pilot landing an airliner) and/or to use scenarios, or one of the “past tense” techniques described below.

Critical Incident Technique

This technique is closely associated with work on safety-critical systems, where system failure is likely to lead to damage, injuries and death. It involves asking the respondent to report on a critical incident in the past. The critical incident is typically used to throw light on boundary conditions such as constraints within which the system has to fit.

This technique can be invaluable because it is grounded in real cases, rather than in hypotheticals. Real cases have a habit of identifying issues which would probably not be caught in scenarios. For instance, real critical incidents often involve an unlikely-sounding combination of things going wrong, all of which are minor or harmless on their own, but which in combination are potentially lethal. Also, real cases often have enough records to let researchers check the “ground truth” of what actually happened, as opposed to what respondents report.

A weakness of this technique is that it depends heavily on the respondent’s memory of what happened. Human memory is liable to numerous biases and errors, so respondents’ reports need to be treated with caution. This can to some extent be handled by cross-checking the reports against contemporary independent evidence.

Hard Case Technique

This technique is usually similar to Critical Incident Technique, but can overlap with think-aloud technique and scenarios.

It involves asking the respondent to report on a particularly difficult case which illustrates one or more key issues from their area of expertise. Usually this is a real past case; the main difference from Critical Incident Technique is that the past case is chosen for its ability to illustrate the key issue(s), rather than for its identification of a significant risk. This can be useful for exploring design rationale, i.e. the chain of reasoning behind a particular design feature, or for exploring expert reasoning more generally.

With past cases, there are the same problems with human memory as were described above for Critical Incident Technique.

This approach can also be used with present-tense hard cases, in which case it can be treated as  form of think-aloud.

It can also be used with future-tense hard cases, in which case it can be treated as a form of scenario.

Summary

The various forms of report complement each other, and other elicitation techniques, neatly. They can get at types of knowledge which are difficult or impossible to reach via other techniques. However, their results need to be treated with caution, because of biases and errors in the human cognitive system; it’s advisable to cross-validate the results using complementary techniques.

We’ll post articles soon describing each of these technique in more detail. We’ll also post an article about the problem of why clients can’t always know what they want (hence the mention of “knowing the unknowable” in the table at the start of this article).

Notes

There is an introductory tutorial about think-aloud technique elsewhere on this blog site.

General notes

Where it’s easy to locate the literature about a topic, we show it in bold italic, and don’t usually give specific references.

Where specific references are needed, and/or where a particular literature is not easy to locate, we usually cite references.

This tutorial is copyleft Hyde & Rugg; you’re welcome to use it for any non-commercial purpose, including academic lectures, provided that you include the “copyleft Hyde & Rugg” with it.


Advertisements

5 thoughts on “Reports: A brief overview

  1. Pingback: Knowing the unknowable, revisited: Why clients can’t know their requirements, and some ways to fix the problem | hyde and rugg

  2. Pingback: Our main posts: An overview by topic | hyde and rugg

  3. Pingback: One hundred Hyde & Rugg articles, and the Verifier framework | hyde and rugg

  4. Pingback: 150 posts and counting | hyde and rugg

  5. Pingback: The Knowledge Modelling Book | hyde and rugg

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s