Finding the right references, part 1

By Gordon Rugg

The best questions are often short.

In a comment on a recent article here, Mosaic of Minds asked which authors I’d recommend for further reading about Likert scales. It’s a fair, sensible question, which lifts the lid on a whole boxful of issues about academic references. Many of those issues are important, but not as widely known as they should be.

This article is the first in an informal series about academic references, online search, and the ways that evidence is used in research. In this article, I’ll be looking at two concepts that provide some useful structure for understanding this general area, namely craft skills versus formalised knowledge, and back versions versus front versions. I’ll start with an overview of these concepts, and then look at the insights they give into different sources of information, including academic references.

Craft skills

Craft skills in the traditional sense are what they sound like; practical, hands-on skills that operate where knowledge meets physical reality. Traditionally, academia had a somewhat snobbish attitude towards craft skills, viewing them as low-level sordid practicalia, very different from the high-status, highly abstract knowledge of Great Thinkers. (Even Great Thinkers, however, had to admit that craft skills were a good thing if the plumbing needed to be sorted out, or the roof was leaking.)

This divide was reflected in the structure of many education systems, which contained a major division between “practical” education and “academic” education. This distinction goes back a long, long time. Craft skills were taught via one set of institutions, such as apprenticeships or the old polytechnics in Britain; formalised academic knowledge was learnt at universities. Different countries used different names for the institutions, but the underlying distinction was widespread.

There was some overlap in content between the two sets of institutions, but the conceptual distinction was deep rooted, as anyone who has worked in a polytechnic turned university will know. The two have very different cultures, values and mindsets.

In more recent decades, researchers have started to realise that the world of formal academic research contains its own craft skills. Once you start investigating this concept as a researcher, you start seeing your own world in a fascinating new way. You get a fresh understanding of a lot of issues that have often been long established for good reasons, but where nobody ever explained to you what those reasons were. A lot of my work with Marian Petre about academic writing and academic reading comes from this approach.

More recently, I’ve been looking at ways of modelling knowledge systematically, tracing it from the most abstract concepts right down to where the ripple of connected concepts bumps into reality. I’ve blogged about it in previous articles:

This work has produced some interesting ways of viewing the old distinction between craft skills and formalised knowledge. Here’s a pair of images showing the core idea schematically.


In the first image, above, the circles and lines are assertions, along the lines of “X is a type of Y” and “Y causes Z” and “A has attribute B”. At the bottom of the diagram, there are assertions that are linked directly to the physical world, such as “sulphur is usually yellow”. If someone asks what you mean by “sulphur” or “yellow” you can point at some physical examples; those examples are real and observable, even if the labels that we use to categorise the examples are intangible social constructs.

If you now represent this schematically, to show the distinction between craft skills and formalised academic knowledge, you get something like the second image, below.


The lowest level assertions are now in the category of “craft skills” and the highest level ones are in the category of “formalised knowledge”.

This visualisation shows the arbitrary nature of the distinction between the two categories. I’ve written before about craft skills, and about how the distinction is not a very helpful one; I argue that it’s more useful to think in terms of how solid the links are between an assertion and observable reality, and in terms of how many links there are between an assertion and observable reality.

When you look at formal academic disciplines from this viewpoint, you soon realise that some involve a great deal of knowledge that links directly to the real world. The physical sciences often involve a lot of field work which is as real as it gets; music and art can also involve a great deal of very specific hands-on knowledge about instruments and materials and processes. That’s very different from a field such as literary studies, where it’s possible to get a degree in the subject without learning anything about the physical process of producing a book or a newspaper.

That’s why I think that the concept of “craft skills” can have some use as a convenient shorthand term, but doesn’t hold together very well if you try to use it more rigorously. For more rigorous use, I prefer to use a representation like the first image, where I can see how many points of contact there are with the physical world within a discipline, and can see which concepts are covered within a particular module or course.

That’s a swift overview of craft skills; I’ll return to the topic later in this article.

Front and back versions

The next diagram shows the concept of front and back versions, applied to the same schematic set of knowledge.


A front version is the version that members a group will show to outsiders; the back version is the behind-the-scenes version that is only shown to group members. It’s a concept introduced by Goffman, who used the analogy of a theatre performance, where there’s the version that’s shown to the audience (the front version) and the backstage reality of performers rushing round changing costume, which is kept hidden from the audience. It’s a useful analogy because it makes the point that the back version isn’t necessarily being kept hidden because it’s shameful or illegal.

As with craft skills and formalised knowledge, this isn’t a clear-cut binary distinction; group membership often involves grey areas and degrees of membership.

That’s an overview of the two sets of concepts at the heart of this article.

What happens when we put them together, and see what light they throw on academic publications?

Applying these concepts to publications


We can represent these concepts as a two by two matrix, as in the image above. This immediately gives us a neat slot for formal academic articles, textbooks, etc; they contain front versions of formalised knowledge, explicitly intended to be available to anyone who wants to read them. It also gives us a neat slot in the bottom left of the matrix for practical manuals, which explain craft skills to anyone who wants to read them.

Those two are easy. What about the other two cells in the matrix, though? That’s where things start to get really interesting.

Like any other field, academia has front versions and back versions. Some back versions, such as the discussions in exam boards, are kept private for good legal reasons involving student confidentiality. Other back versions involve issues such as acceptable versus unacceptable shortcuts in research methodology, which are more about conventions and norms. Still other back versions involve scandals, potential scandals, and suspicions, ranging from mild personal disapproval to reports of criminal misbehaviour.

So how do these appear in publications, if at all?

There have been several high-profile cases over the years where high profile classic research findings have been called publicly into question. Examples include Burt’s work on IQ and Meade’s work on teenage sexual behaviour in Samoa. A common pattern is that there are suspicions in a research field about a particular set of findings for years, followed by published claims that there is evidence showing that the original data behind those findings shouldn’t be trusted. In some cases, there’s evidence that some or all of the original data had been fabricated; in others, there’s evidence of sloppiness or data manipulation by the researcher and/or by the research assistants carrying out the data collection; in others again, there’s evidence that the human participants gave misleading information to the researcher, either maliciously or as a joke or because of cultural norms about it being polite to tell people what they want to hear.

Some of these cases involve formalised academic knowledge (e.g. in cases of deliberate fraud); others involve craft skills in data collection and/or data analysis. That aspect of the two by two matrix holds up reasonably well.

The matrix doesn’t hold up so well in regard to that throwaway statement about “published claims”. If a claim is published in a journal or book that’s available for anyone to read, then it should count as a front version. That’s how a lot of re-assessments of debatable claims start out, as published academic articles, in the public domain. However, it can take years before those claims reach the non-specialist world.

In reality, there’s a huge difference between something being available for anyone to read, and anyone being able to read and understand it. Academic technical language is often effectively incomprehensible to outsiders because it’s dealing with concepts and terms that take years to learn. This means that we need to make a distinction between front versions that are actually understandable by non-specialists, and front versions that aren’t understandable by non-specialists.

A three way model

This gives us a matrix with six cells. There’s one row for formalised knowledge and one row for craft skills. There’s one column for “plain English” front versions, plus a second column for “specialist English” front versions and a third column for back versions.

When we plot various types of publication on this matrix, it gives some useful insights.

publication types

This diagram looks complicated, but it’s actually quite simple if we go through it step by step.

Plain English front versions

On the left, we have the “plain English” front versions.

Some of these are completely accessible to non-specialists (e.g. newspaper articles). Some are quite specialist and deal with abstract knowledge, but are still intended for the general educated reader (e.g. popular science magazines). Some are quite specialist and focus on craft skills (e.g. specialist hobby magazines).

Academic language front versions

In the middle, we have the front versions that are written in specialist language. Specialist language ranges from fairly accessible to practically incomprehensible to non-specialists. Textbooks are usually on the “fairly accessible” side, whereas journal and conference articles are usually on the “practically incomprehensible” side.

I’ve included tutorial articles in this column, at the area of overlap between craft skills and formalised knowledge. Tutorial articles are invaluable if you’re trying to learn a new method to use in your research. They are what they sound like, namely articles that give you a tutorial on how to use a particular method. Although tutorial articles are invaluable, most students and new researchers have never heard of them.

I’ve also included specialist monographs in this column. These are similar to journal articles in that they’re intended for a readership of specialist researchers, but different from journal articles in that they’re a single publication. Usually, but not always, they’re a book. (For brevity, I won’t go into detail about the fine points of this.) Some are classic texts that every specialist in the relevant field will know well, and probably own; others will only ever be read by a handful of people (but might still be classic texts to that handful of people).

Academic language back versions

I’ve treated this column differently from the other two. The types of publication I’ve cited are publications about the back version of research, rather than publications that are the back version of research. I’ve done this because most readers of this article will find those publications more useful and/or thought provoking than the others. (If you’re wondering about what would count as publications that are the back version of research, then examples include original field notes and personal notebooks, which are often legally required to be kept confidential.)

There’s been a quiet growth over recent decades in research into how research is actually conducted. The sociology of science and the history of science are both well-established research fields, and are usually involved in heated debates about assorted topics. Some of these debates are about highly formalised meta-knowledge; examples range from postmodernism to the debate between frequentist and probabilist models in statistics. Others go deep into the craft skills level, such as debates about the role of local interpreters and guides in what anthropologists get to hear from local people.

If you’re considering a career in research, or you’re a new researcher, it’s well worth dipping into these literatures; they give a useful grounding in the realities on which research is based, and on the questions that arise from those realities.

Concluding thoughts

Returning to the question at the beginning of this article, if you’re looking for references on a topic, then the answer to a “what would you recommend?” question is: “It depends”.

That might sound unhelpful, but it’s actually helping the person asking the question to refine their understanding, and to plan their reading more efficiently.

For instance, if you’re completely new to the concept of Likert scales, then the best place to start will be at the left of the diagram above, with plain English versions. The advice that I usually give students in this situation is to find a textbook that has a writing style they’re comfortable with, and that has a chapter on the topic. The reasoning behind this is that textbooks are usually a fairly trustworthy source for an overview, and that a chapter will usually be short and simple enough to be easily understandable; an entire textbook on Likert scales will probably drown the average reader in detail.

If you’re writing about previous good examples of using Likert scales, then you’ll want to track down some classic journal articles on the topic (I’ll be writing about how to do this in a separate article).

If you want some tips on the low-level practical issues involved in using Likert scales, then a tutorial article is a good place to start.

If you’re wanting to pick up some marks for showing a deeper critical understanding of the strengths and weaknesses of Likert scales, then you’ll need to go into the literature in the rightmost column, including the theoretical literature on research methods (e.g. issues involving psychophysics and measurement theory in relation to Likert scales) and the craft skills literature on research realities (e.g. whether some cultures believe that it’s polite to respond in a particular way to Likert scales).

So, in conclusion, a good, clear, short question led into a long answer. So it goes, sometimes. I hope that you’ve found this useful; I’ll be writing about other aspects of literature, references and search in later articles.


You’re welcome to use Hyde & Rugg copyleft images for any non-commercial purpose, including lectures, provided that you state that they’re copyleft Hyde & Rugg.

There’s more about the theory behind this article in my latest book, Blind Spot, by Gordon Rugg with Joseph D’Agnese:

My books with Marian Petre:

A Gentle Guide to Research Methods:

The Unwritten Rules of PhD Research:






7 thoughts on “Finding the right references, part 1

  1. Pingback: Finding the right references, part 2: Getting a first overview | hyde and rugg

  2. Pingback: Finding the right references, part 3: Breadth, depth and the T model | hyde and rugg

  3. Pingback: Finding the right references, part 1 | Research...

  4. Pingback: Doing online searches | hyde and rugg

  5. Pingback: The Knowledge Modelling Book | hyde and rugg

  6. Pingback: Critically reviewing the literature, the quick and dirty way | hyde and rugg

  7. Pingback: Getting an overview of the literature via review articles | hyde and rugg

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s