Truthiness, scienciness, and product success

By Gordon Rugg

This is a Tempest Prognosticator.

800px-Tempest_Prognosticator“Tempest Prognosticator” by Badobadop – Own work. Licensed under CC BY-SA 4.0 via Wikimedia Commons – https://commons.wikimedia.org/wiki/File:Tempest_Prognosticator.jpg#/media/File:Tempest_Prognosticator.jpg

It’s a splendid example of nineteenth century ingenuity, right down to the name. What does it do? It’s intended to let you know if a storm is approaching. The way it does this is as wonderfully nineteenth century as the name. The Prognosticator is operated by twelve leeches, each of which lives in a bottle. When storms are approaching, the leeches become agitated, and climb out of the bottle. When they climb out, they disturb a piece of whalebone, which activates a bell. The more serious the risk of storm, the more leeches climb out, and the more bells ring.

When it was invented, in the 1850s, the British government were looking for better systems of weather forecasting. They were particularly interested in predicting storms, which were a major threat to sail-powered ships.

So, did they adopt the Tempest Prognosticator as their preferred solution to this problem? As you might have guessed, they didn’t. Instead, they opted for the FitzRoy’s Storm Glass.

320px-Storm_Glass_FitzRoy_Sturmglas“Storm Glass FitzRoy Sturmglas” by ReneBNRW – Own work<<<Selbst erstelltes Foto.. Licensed under CC0 via Wikimedia Commons – https://commons.wikimedia.o/wiki/File:Storm_Glass_FitzRoy_Sturmglas.JPG#/media/File:Storm_Glass_FitzRoy_Sturmglas.JPG

The Storm Glass also had a wonderfully nineteenth century name, but it worked on a very different principle. (In fairness, almost anything would work on a different principle from leech power…)

The Storm Glass was a glass that contained liquid, with chemicals dissolved in the liquid. Different versions of the glass used different ingredients, but the core concept was the same. The contents of the glass were sensitive to weather conditions, and showed different levels of crystal formation within the glass, depending on the weather.

The British government adopted this device as the preferred form of storm warning, and distributed storm glasses to a number of coastal towns in 1859, when there was a spate of violent storms.

And there the story would end, if you were just looking for examples of Victorian eccentricity. If, however, you’re looking for the underlying factors in designing successful new products, there’s a lot more story to tell, and the full version of the story has some powerful implications.

The bottom line

For a start, there’s the awkward consideration that the Storm Glass didn’t actually perform at above chance levels in predicting storms. The crystallisation rates correlated with temperature, but not with impending storms. In short, it didn’t work.

Nobody knows for sure whether or not the Tempest Prognosticator actually could predict storms, but its underlying assumption is far from silly; various types of animal are sensitive to weather conditions. The Tempest Prognosticator might quite possibly have worked.

What was going on?

It might seem reasonable to assume that when lives are at stake, people will check whether or not the relevant technology actually works. Reasonable, yes. True, no. Surprisingly often, technologies are widely adopted even though they either don’t work at all, or make things worse.

In an ideal world, people would check the relevant evidence thoroughly and systematically. However, it’s not an ideal world. One simple, powerful issue is lack of time. There isn’t enough time to check all the evidence that relates to every decision that you make.

Instead, human beings use a lot of heuristics – rules of thumb that don’t always work, but that work often enough and swiftly enough to let you make reasonably good decisions reasonably often.

That was probably a major factor in the British government’s decision to adopt the FitzRoy Storm Glass. FitzRoy wasn’t just anybody; he was Vice Admiral Robert FitzRoy, RN. He was the FitzRoy who commanded the Beagle during Darwin’s famous journey; he was the FitzRoy who set up the first incarnation of the Meteorological Office in 1854; he was the FitzRoy who was Governor of New Zealand from 1843 to 1845.

That’s a pretty strong portfolio.

In contrast, the Tempest Prognosticator’s inventor, Dr Merryweather, was honorary curator of the Whitby Literary and Philosophical Society’s Museum. That doesn’t inspire so much confidence. Also, his invention was inspired by a couple of lines of poetry about a leech rising in his prison; again, this isn’t the strongest of arguments for adopting his technology.

When you look at the bigger picture, you soon discover that FitzRoy’s poor performance with the Storm Glass was an exception, rather than a rule. Other devices that he invented or improved, such as various forms of barometer, were a lot more successful, and many of his innovations are still in use today. So, overall, the heuristic of choosing the candidate with the stronger relevant track record actually performed reasonably well in the broader context, and just came off the rails with the one example of the Storm Glass.

Truthiness, scienciness and covering your back

Looking at track records is one heuristic for swiftly getting a rough idea of how much to trust an idea.

Another heuristic is to look at the language being used to describe it. This is rich ground for charlatans, many of whom play to this heuristic by describing their wares using words that look scientific, and stories that appear to have the ring of truth. The terms “scienciness” and “truthiness” were invented to describe this phenomenon. It’s depressingly widespread.

Another factor that is probably at play in the story of the Tempest Prognosticator is the simple principle of covering your back if you’re a bureaucrat making a decision. There used to be a saying that nobody ever got fired for buying IBM. Similarly, a civil servant probably wouldn’t get fired for selecting a Storm Glass invented by Vice Admiral Robert FitzRoy, RN, Meteorological Statist to the Board of Trade. Selecting a leech-powered device invented by a museum curator, inspired by two lines of poetry, wouldn’t be such a good career decision.

So, at one level, this story makes good sense.

At another level, though, there’s the nagging question of whether the Tempest Prognosticator actually worked.

I’m fascinated by the faulty assumptions made at key decision points in science. When the Tempest Prognosticator was pitted against the Storm Glass, science was making huge progress in chemistry. The Storm Glass fitted comfortably within the schema of chemistry being the way of the future. The Tempest Prognosticator, in contrast, was based on leeches, which had associations with an older, unscientific world. The Tempest Prognosticator was on weak ground with regard to the heuristic of “Modern is better than old”.

As far as I know, the Tempest Prognosticator was never systematically evaluated in terms of how well it actually worked. Instead, it appears to have been rejected on the basis of heuristics. The history of science is littered with tantalising examples of innovations which were ignored for decades or centuries because of similar decisions, but which turned out to have been right all along.

Having said which, honesty compels me to admit that I don’t have any plans to acquire a Tempest Prognosticator in the foreseeable future. Sometimes, heuristics end up with a sensible conclusion, even if the way they got there was a bit dodgy…

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s