The quick and dirty approach to meta-analysis

By Gordon Rugg

In an ideal world, everyone would always do everything perfectly. However, it’s not an ideal world.

So what can you do when you’re trying to make sense of a problem where there’s conflicting evidence, and you don’t have time to work through all the relevant information?

One approach is simply to decide what your conclusion is going to be, and then to ignore any evidence that doesn’t fit. This is not terribly moral or advisable.

Another is to do a meta-analysis, to assess the quality of the evidence as a whole. This sounds impressive; it also sounds like hard work, which it is, if you do a full-scale proper meta-analysis. Most academic researchers therefore use two types of meta-analysis.

  • The first is the quick and dirty type, which normally gives you a pretty good idea of whether the topic is worth spending your time on.
  • The second is the proper type, which is time-consuming, and requires sophisticated knowledge of research methods, including statistics.

This article, as the title subtly implies, is about the quick and dirty approach. It’s a flawed, imperfect approach, but it’s a good starting point in a flawed, imperfect world.

Continue reading