The Verifier approach: part1

By Gordon Rugg

The central theme in Blind Spot, the new book by myself and Joe D’Agnese, is the Verifier approach. The Verifier approach is a way of spotting errors in human reasoning; in particular, spotting errors made by experts dealing with long-standing problems where it looks as if the experts have ground to a halt and can’t see where to go next.

Its starting point is simple. Human beings make pretty much the same types of mistakes, regardless of their level of expertise, or of the field in which they’re working. If you know what those types of mistakes are, then you can go hunting for them.

That idea has been around for a while. What’s new about Verifier is that it tackles human error using a bigger, broader set of methods than previous approaches. For example, traditional formal logic can tell you whether someone’s chain of reasoning is logically sound, but it doesn’t give you any guidance about how to check whether a particular assumption in that chain of reasoning is actually true or not.

So, what we’ve done with Verifier is to assemble virtual toolboxes for each of the key stages in looking for errors. They’re not physical toolboxes or software toolboxes; they’re collections of methods.

There are three toolboxes for spotting errors, plus a fourth toolbox which is about preventing or reducing errors. The key concept they have in common is that each toolbox has a set of evidence-based guidelines for choosing the most suitable method for a particular purpose. We’ve erred on the side of inclusion when drawing up the toolboxes, so they tend to include large numbers of methods, even though they don’t claim to be exhaustive collections.

This article introduces one of those toolboxes; we’ll look at other toolboxes in later articles.

In practice, we’ve found that within each toolbox, there are a few methods that are invaluable, and others that we rarely use. For example, we’ve found graph theory invaluable, particularly in the forms of laddering and facet theory. We also make extensive use of the concepts of sequential processing and parallel processing, particularly pattern-matching. Most of the methods we’ve found invaluable aren’t very widely known outside their home territory, so we’ll be writing about them in future articles.

In addition to the toolboxes, Verifier has an overall process, which deals with which toolboxes to use when, and how. That process doesn’t follow neat clearly-defined steps, for various good reasons. The full story is more complex than the seven-step method that you may have seen. We’ll discuss the process after the articles about the individual toolboxes.

Toolbox 1: Elicitation

The first toolbox is about how to gather accurate, reliable information to feed into the logical testing. This is difficult, particularly when you’re dealing with information that people find difficult to put into words, or involving topics that people are reluctant to talk about. Our underyling approach here is based on the ACRE framework developed by Neil Maiden and myself. In short, it divides knowledge, memory and communication into various types and sub-types, and then maps each of those types onto a list of methods for eliciting information, identifying the appropriate elicitation method or methods for each type.

The image below shows a schematic representation of this framework.


Each of the knowledge types is subdivided, as follows.


Each of these subdivisions is mapped onto appropriate elicitation types, as shown below.

Knowledge, memory and communication types (left column) and elicitation methods (right column)


The image above is a simplified version – for example, there are numerous sub-groups of the elicitation methods above, such as structured, semi-structured and unstructured interviews. Also, the categorisation of elicitation methods can be significantly improved by focusing on the component parts within each method, as in the paper on method fragments by Neil Maiden, Peter McGeorge and myself. However, the diagram gives a sense of the underlying concepts in use.

The colouring of the boxes on the left side of the diagram is intended to indicate the progression from explicit knowledge, readily accessible via introspection, through to completely tacit knowledge. The green of “future knowledge types” is intended to indicate that the contents of these boxes are qualitatively different from the contents of the boxes which contain actual memories and knowledge, rather than guesses about future actions and preferences.

The ACRE paper describes the issues and the mappings; we’ll be blogging about this in more detail in further articles.

That’s a brief overview of the elicitation toolbox. Its key points are that it’s big, but of a tractable size, and that it has a guiding framework derived from the literatures on memory, communication and skill.

We have tutorial articles on some of the techniques mentioned in the framework, including laddering, on the Hyde & Rugg website. More articles are on the way.


The ACRE framework reference is:

Maiden, N.A.M. & Rugg, G. (1996). ACRE: a framework for acquisition of requirements.

Software Engineering Journal, 11(3) pp. 183-192

The method fragments paper reference is:

Rugg, G., McGeorge, P. & Maiden, N.A.M. (2000). Method Fragments. Expert Systems  17(5), pp. 248-257

A tutorial on card sorts:

Rugg, G. & McGeorge, P. (2005). The sorting techniques: a tutorial paper on card sorts, picture sorts and item sorts. Expert Systems, 22(3) (NOTE: this is a reprint of our 1997 paper in Expert Systems)


12 thoughts on “The Verifier approach: part1

  1. Pingback: Knowing the unknowable, revisited: Why clients can’t know their requirements, and some ways to fix the problem | hyde and rugg

  2. Pingback: Our main posts: An overview by topic | hyde and rugg

  3. Pingback: An education framework based on knowledge modelling | hyde and rugg

  4. Pingback: Compiled skills and education theory | hyde and rugg

  5. Pingback: How complex should education theories be? | hyde and rugg

  6. Pingback: One hundred Hyde & Rugg articles, and the Verifier framework | hyde and rugg

  7. Pingback: The “compass rose” model for requirements gathering | hyde and rugg

  8. Pingback: How long is an education good for? | hyde and rugg

  9. I was doing some research in to ‘knowledge management’ recently and I have noticed similar themes as you expressed in your post around the same time period. For example the work by Kane et al (2006) who explores research methodologies used in knowledge management and expressed that tacit and explicit information are the foundations of understanding systems.

    Anyway, to the point: Would you say that the ACRE framework can be used as a knowledge management tool?

    My interpretation of knowledge management is: decision making and problem solving leading to change


    • Good question. Yes, I think that the ACRE framework and related concepts such as parallel versus sequential processing are highly relevant to knowledge management.

      I think that knowledge management has been trying to tackle the right problems, but hasn’t had all the tools it needed. For instance, just using a two-way distinction between tacit information and explicit information isn’t sufficiently fine-grained. When you start using a finer-grained set of distinctions, like the ones in the ACRE framework, then you can get much better traction on problems.

  10. Pingback: How complex should models of education be? | hyde and rugg

  11. Pingback: Creativity and idea generation | hyde and rugg

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s