Literature reviews

By Gordon Rugg

Almost every academic article begins with a literature review. As is often the case in academia, this is a rich, sophisticated art form, whose complexities are often invisible to novices. As is also often the case in academia, there are usually solid, sensible reasons for those complexities. As you may already have guessed, these reasons are usually not explained to students and other novices, which often leads to massive and long-lasting misunderstandings.

This article looks at the nature and purpose of literature reviews. It also looks at some forms of literature review which are not as widely known as they should be.

It’s quite a long article, so here’s a picture of a couple of cats as a gentle start.

First, I’ll look at a distinction which looks like a nit-picking piece of pedantry, but which actually goes to the heart of what literature reviews are about.

Literature reports versus literature reviews

One frequent piece of feedback to students from markers at university level is along the lines of “This is a literature report, rather than a literature review”.

What’s the difference, and why does it matter?

A literature report is what it sounds like. It tries to report what’s in the literature, without any explicit attempt to comment on or critically assess that literature.

This might sound like praiseworthy objectivity. The reality is different. Most academic literatures are huge; for some disciplines, it’s physically impossible to read all the relevant publications within a human lifetime. So you have to make judgment calls about what to include and what to exclude. This, by definition, means that you can’t report impartially; you have to make judgments about what matters and what doesn’t. So, literature reports are often the result of naiveté or of disingenuousness.

A classic example of this going wrong is “he said/she said” journalism, which often results in false equivalences, where the journalist gives equal weight to one argument based on a massive amount of well-established research and to another argument based on the gibberings of someone in a tinfoil hat.

There’s a second problem with the literature report, if you’re a student or new researcher. Researchers need to make critical assessments of previous research into a problem, and to identify errors and omissions in the previous research. You can’t do this simply by reporting what’s been written previously. Instead, you need to review the literature. That’s a big difference.

This often causes cross-cultural problems. In many cultures, it’s the height of presumptuousness for a student to criticise the work of the Great Masters in the field. This involves disciplinary cultures as well as national cultures; for instance, in popular debate about the humanities in the UK and the USA, there’s a tendency for people to invoke the names of Great Thinkers like Plato as if those ancient opinions were revealed truths that will stand for all time. I’ve written about this as the “charm bracelet” model of referencing in an earlier article.

The traditional literature review

So what does a literature review look like?

In most disciplines, it involves a historical overview of how the relevant field has developed up to the present day.

For instance, if you’re writing about the impact of biology research on education theory, your literature review would start with the earliest writers on education theory who drew on biology. You would then move forwards through time, identifying key developments, key researchers and key publications, with a critical evaluation of each one in turn, until you reach the present day.

The core concept is that simple. However, you rapidly run into potential pitfalls and dead ends.

Cherry picking

One classic pitfall is cherry picking. This involves selecting only evidence that supports your argument, and ignoring evidence that undercuts or demolishes your argument.

Cherry picking is common. That doesn’t make it any less bad.

A proper literature review will give a fair, balanced overview of the weight of evidence for the key relevant arguments about each of the points covered in the literature review.

A cynical view is that this is impossible, since “fair and balanced” involves subjective value judgments. A common cynical follow-up argument is that since “fair and balanced” is impossible, then anything goes, so we might as well openly cherry pick. This sounds superficially plausible, but as with Zeno’s paradox, the reality is different. As in the example above, you can reach some level of objective agreement that the second tree has a higher number of ripe cherries than the first, even if you don’t know the exact number. I’ll return to this issue repeatedly through the remainder of this article.

Doing a “good” literature review: Subtexts and reasons

So far, I’ve mainly focused on problems that can arise when doing literature reviews. In this section, I’ll look at current best practice, and I’ll unpack some of the subtexts and rationales involved.

Arguably the most extreme and demanding form of the literature review occurs in a PhD. Some quick clarification: I’ll sometimes distinguish between the dissertation, which is the written document, and to the thesis, which is the theoretical argument being described in the dissertation.

When you write up your PhD, you don’t have much room for evasion in your literature review. You can’t hide behind the excuse of a tight word count, as you can when writing a journal article; PhD dissertations do have length limits, but those limits still give you more than enough space for a thorough literature review. You can’t hide behind the excuse that you’re writing a simplified version for a specific readership; the whole point of the PhD is that you’re writing the full-strength version for professionals who are leading experts in your chosen field. You have to do it so well that it will stand up to sustained hard, well-informed questions from the external examiners.

What will the external examiners be looking for? Various things.

One thing that they won’t be particularly concerned about is whether you disagree with orthodoxy. The conspiracy theory that academics silence dissent from academic orthodoxy is a caricature of reality. A classic way of making your reputation in academia is to find a point where orthodoxy is wrong. If you do that for a point where orthodoxy is very wrong, you can get a Nobel Prize.

One thing that the examiners will be looking for, however, is how well your literature review summarises the previous literature on your chosen topic. “Well” in this context means things such as:

• Completeness of coverage – have you identified all the key findings, issues, researchers and publications?
• Currency – is your review completely up to date, or have you missed any significant recent findings?
• Accuracy – does your review accurately reflect what is being said in the previous literature?

If you know what you’re doing when writing up your PhD, then you make sure that your literature review is exhaustive in its coverage, and that you include at least one reference to a publication from within the last twelve months, and that you include at least one detailed discussion of wording within a key publication to show that you’ve read the literature thoroughly, rather than relying on second-hand descriptions from Wikipedia. All these things send out the right signals to the examiners.

Conversely, if you’ve failed to mention key publications etc relating to your thesis, then the examiners will view this as a major problem, and will start asking some very hard questions. For instance, if your thesis is based on a model of how memory operates when learning material in a classroom environment, and you haven’t mentioned key figures such as Baddeley, then this raises serious questions about your competence. It’s your job to show that you’ve read the relevant literature; the examiners’ default assumption is that if you haven’t mentioned a relevant concept, it’s because you’ve failed to learn about it. If you demonstrate in the viva that you have in fact read about Baddeley (for instance) but didn’t think that his work was relevant to your thesis, you’re still in deep trouble, because this is an incredibly amateurish way to handle the issue. What you should do in such circumstances is to explain why Baddeley (or whoever) appears at first sight to be relevant, but actually isn’t relevant.

Framing, arguments and theses

The discussion above raises a deep question. How do examiners, or other researchers, decide what the key issues and concepts etc are within a field? There isn’t a single perfect answer. There are various answers, none of which is perfect, but which together at least manage to reduce or eliminate a lot of potential problems.

One criterion for assessing what to include is professional consensus within the field. It’s far from perfect, but it’s a good start. You should get a fair idea of professional consensus from reading the literature; you’ll soon start seeing the same names occurring over and over again in the literature reviews within the articles that you read.

This is one of many places where peer review plays an important role. Although peer review is far from perfect, it’s still an invaluable criterion for deciding what’s worth considering seriously, and what isn’t. The popular literature can be, and often is, written by amateur idiots who think that they know more about a topic than people who study it for a living. The peer reviewed literature, on the other hand, has to pass at least some quality control before it appears in print, so it’s a good place to start.

You should also get useful information about professional consensus from review articles, whose purpose is to give an overview of key research developments in a field over recent years (usually about the last ten to twenty years). These are an invaluable resource for new researchers, since they give a synoptic overview of the field, and they’re usually written by major figures within the field. You may not agree with the opinions of the person writing the review article, but you can usually be fairly confident that they will have identified the most significant concepts, findings and publications.

One problem with the traditional literature review, as described above, is that it can easily nudge a field towards a particular framing or agenda, which can take the field off in a plausible but wrong direction.

This is one reason that I have mixed feelings about invocations of Great Names such as Plato and Rousseau in literature reviews. On the one hand, yes, they might have prefigured some current debates. On the other hand, they were operating in a world very different from ours, long before modern science got going, so their opinions will probably be only of historical interest. If we let Plato’s views influence the agenda for a research question, then we’re accepting the agenda of someone writing before the discovery of gunpowder, of America, of chemistry, of electricity and of modern mathematics, to name an arbitrary handful of issues.

This is one of the points raised by some early postmodernists. They asked questions about why some topics are heavily researched, but not others. The answer often comes down to social choices and social status; topic A is viewed as prestigious and proper, and topic B is viewed as low-status and trivial. For instance, researching Jane Austen’s work is viewed as completely acceptable; researching Barbara Cartland’s work is viewed as “junk research” or trendy nonsense.

Systematic Literature Reviews

There have been various attempts to approach literature reviews more systematically.

At the level of academic disciplines, fields such as sociology and linguistics and media studies have a long and honourable tradition of tackling head-on the role of social status and social values within research framing. Studying French from within linguistics, for example, is a world away from studying it from within French Studies.

At the level of specific research methods, one approach that has recently received increasing attention is the Systematic Literature Review (SLR). This is a method which produced some very useful results in medicine, where researchers used it to gain an overview of the pattern of research findings about a particular topic – typically how well or badly a particular intervention performed for a particular medical condition.

This approach involves careful, detailed documentation of the process used in the search, including the choice of keywords and of bibliographic database. It looks very scientific. However, it has some serious limitations, which I’ve discussed in another article. It works well if you’re dealing with crisp sets, i.e. entities that can be clearly and unambiguously defined, such as a particular medication and a particular illness that can be diagnosed via an objective test. It doesn’t work well with fuzzy sets, where the entities are not clearly defined or objectively measurable.

Another drawback of SLRs is that they only give one type of insight into the literature. They don’t, for instance, show what happens when you plot the strength of a claimed effect against the quality of the publication where it’s described.

Similarly, SLRs don’t give any insights into developments across time within a field. They can tell you how many articles appeared on a particular topic in successive years, but they can’t by themselves explain why the number of articles on a topic changed or remained constant.

Argumentation and literature reviews

Some of my recent work with various colleagues has involved using visual representation of argumentation within literature reviews. This provides a clean, systematic framework for a chain of reasoning, and for choosing topics to include within a literature review.

For example, the image below shows what might happen if you trace the successive causes of a condition across two different literatures that don’t normally interact much with each other (e.g. the literature on Ehlers-Danlos Syndrome and the literature on podiatry). The hypothetical example below shows in white boxes the causes of condition X that have been identified within one field, and shows in green the contributory causes that have been identified within a different field.

This can be particularly useful for identifying weaknesses in one literature that might be fixed by importing concepts from another literature. This is a well established approach within research; I’ve written about it in an earlier article, using the semi-humorous analogy of the magic weapon that the protagonist uses to tackle the problematic dragon.

Closing thoughts

That’s a brief overview of some key concepts relating to literature reviews. Literature reviews are an extremely important part of research, and there’s a lot more that can be said about them.

If you’re interested in this topic, you might like the links below. They’re not an exhaustive collection; they’re just a taster, but they’re interesting tasters.

With regard to topics that aren’t addressed within a research literature, a recurrent problem is the significant absence, i.e. something whose absence isn’t an accident. The difficulty is deciding what is significant and what isn’t, since in principle anything could be viewed as a significant absence, depending on the opinion of the individual.

A classic example is the way that protagonists in action stories often don’t have a mother. There’s an insightful article about this topic here:

http://www.theatlantic.com/magazine/archive/2014/07/why-are-all-the-cartoon-mothers-dead/372270/

If you find that article interesting, then you’ll probably love the TV tropes site:

http://tvtropes.org/pmwiki/pmwiki.php/Main/HomePage

It can, however, become horribly addictive, and can transform your experience of TV viewing…

Returning to the old school: The field of literary criticism is traditionally very text-based, with comparatively little use of graphical or quantitative approaches (apart from stylometrics, which is arguably a separate field).

One exception is the work of Moretti, who used a variety of methods to examine texts graphically and quantitatively.

http://www.amazon.com/Graphs-Maps-Trees-Abstract-Literary/dp/1844670260

A great deal more could be said about literature reviews, and has been said in numerous other places by numerous other people, but it’s a hot July day and my admin stack is calling, so I’ll end here.

Notes

As usual, I’ve used bold italics for useful concepts that are covered well in easily accessible sources such as Wikipedia.

You’re welcome to use Hyde & Rugg copyleft images for any non-commercial purpose, including lectures, provided that you state that they’re copyleft Hyde & Rugg.

There’s more about the theory behind this article in my latest book, Blind Spot, by Gordon Rugg with Joseph D’Agnese:

http://www.amazon.co.uk/Blind-Spot-Gordon-Rugg/dp/0062097903

There’s a lot more about academic writing, including literature reviews, in my books with Marian Petre, in particular The Unwritten Rules of PhD Research:

Related articles:

https://hydeandrugg.wordpress.com/2014/07/11/representing-argumentation-via-systematic-diagrams-part-1/

https://hydeandrugg.wordpress.com/2014/07/05/logic-evidence-and-evidence-based-approaches/

https://hydeandrugg.wordpress.com/2014/06/25/the-quick-and-dirty-approach-to-meta-analysis/

https://hydeandrugg.wordpress.com/2014/06/01/cherry-picking-and-dodgy-reasoning-for-beginners/

https://hydeandrugg.wordpress.com/2014/03/21/academic-writing-and-fairy-tales/