Smoking-Bun

I’ve started reading The Wisdom Of Crowds, James Surowiecki’s critically acclaimed tome from 2004. It’s subtitled “Why the many are smarter than the few” and the basic thesis of the book is that when a crowd’s opinions are taken as a whole, they are often more accurate than the opinion of one learned person who may just be wrong.

Renowned playwright Tom Stoppard calls it ‘brilliant and fascinating’ on the back cover. But he’s not a crowd, he’s one person. And many people are smarter than the few. It seems odd to see his individual quote on the back of the book – surely a group of playwrights would be a better judge.

The first case study in the book is from British scientist Francis Galton (one person), who studied the group results of the ‘Guess the weight of the Ox’ competition at a regional country fair…

Galton undoubtedly thought that the average guess of the group would be way off the mark. After all, mix a few very smart people with a some mediocre people and a lot of dumb people, and it seems likely you’ll end up with a dumb answer. But Galton was wrong. The crowd had guessed that the ox, after it had been slaughtered and dressed, would weigh 1,197 pounds. After it had been slaughtered and dressed, the ox weighed 1,198 pounds. In other words, the crowd’s judgement was essentially perfect.

Next case study. The US submarine Scorpion sinks somewhere in the North Atlantic, with no technology available to accurately pinpoint its position. A naval officer John Craven (one person) comes up with a way of assessing a group of experts’ varying predictions on where the submarine may lay…

The location that Craven came up with was not a spot that any individual member of the group had picked. In other words, not one of the members of a group had a picture in his head that matched the one Craven had constructed using the information gathered from all of them… Five months after the Scorpion disappeared, a navy ship found it. It was 220 yards from where Craven’s group had said it would be. What’s astonishing about this story is that the evidence that the group was relying on in this case amounted to almost nothing. It was really just tiny scraps of data.

One more quick one, just to finally put the issue to bed…

A classic demonstration of group intelligence is the jelly-beans-in-the-jar experiment, in which invariably the group’s estimate is superior to the vast majority of the individual guesses. When finance professor Jack Treynor ran the experiment in his class with a jar that held 850 beans, the group estimate was 871. Only one of the fifty-six people in the class made a better guess.

You convinced now? Do you now understand ‘why the many are smarter than the few’? It’s a glorious confirmation of democratic principles, of society, of human being’s one-ness. Or it’s a crock of shit.

What if I were to say that these these examples were simply an illustration of ‘good luck’. Or rather a realistic expectation of a small number of cherry-picked examples to support a certain conclusion. Obviously I am not a crowd, so my judgement isn’t to be trusted. Is any one of these examples a proper experiment, that is even capable of proving this point about the accuracy of group judgement? Of course not.

Let’s imagine a hundred Francis Galton’s going to a hundred country fairs with a hundred Oxes and examing a hundred sets of ‘guess-the-weight’ results. Now if in all of those results the crowd judgement was ‘essentially perfect’ then you have a bit of a phenomena – indeed it might be the most amazing phenomena ever observed in sociological history. But let’s pretend that only one out of a hundred times was the group accurate – that would be the only one we’d hear about in this book. The other 99 Francis Galton’s wouldn’t consider their results to be worthy of any further analysis. My guess is that’s what happened – we’ve heard about the only time the results fit this claim about crowds. Yes it’s just a guess, but it is just as likely as the guess inspired by Galton’s presented experience here.

There are other problems here. It is assumed, for example, that those guessing the weight of the ox were a perfect cross-section of society, rather than a specialised group with a professional interest in the matter, ie the sort of people who might be wandering around a country fair looking at livestock. The writer notes that a number of the guesses were discarded as unintelligible, so it’s only a judge of the wisdom of crowds that have mastered basic writing skills. Imagine how the results might be altered according to the way in which the guesses were made – in small groups perhaps,  influencing each others guesses and colouring the experiment in different ways. Could the guessers see the guesses already made? – this is absolutely crucial to the test!!!

A submarine sank and a ‘group judgement’ was within 220 yards of the actual position of the vessel. Was this a control experiment? No it was a one-off – submarines don’t sink every day. Presumably if one expert had been proven to be bang-on with their educated guess of where the sub was, we’d be reading about their phenomenal performance in a book called ‘The Wisdom of One Individual’. What is hilarious about this example is that it is admitted that there is almost literally no information to work on – unless we believe in some sort of extra-sensory-perception being employed, the result is quite clearly blind luck.

This submarine example isn’t a crowd at all, it is a presumably hand-picked group of experts of unspecified number – if there were only 3 of them for example it totally changes the implications of the results. As noted, these experts were asked to guess on all sorts of things about the sub, the possible problems it encounters, its speed and angle of descent – presumably the group was accurate on all of these things as well, and the writer has just forgotten to tell us. If the group average of the guesses of the angle of descent was way off for example, how smart are the many then?

I included the last ‘experiment’ because of the hilarious line “invariably the group’s estimate is superior to the vast majority of the individual guesses”. Because if you think about it, whatever the spread of the guesses, presuming that there is a reasonable difference between them, the average is likely to be more likely to be reasonable, and also different (or ’superior’) to the majority. That’s the whole bloody point of taking an average.

Funny in this case that the writer claims that ‘invariably’ this jelly-bean experiment works, but strangely this is one claim that they don’t have accurate data for – what a shame, as it’s the only claim which might genuinely have proven their point!

This is not even a measure of the general wisdom of crowds – it is the (supposed) specific wisdom of crowds in certain situations, ie ones where there is a large scale on which to guess, where the actual answer isn’t an unusual ‘outlier’, and where a spread of wrong guesses can be easily averaged into a ‘group guess’. Ask a crowd what temperature magnesium melts at, and see if your average is more accurate than the guesses. Some will know and the rest will guess – it’s as simple as that. If only all judgements were over jelly-beans in a jar, rather than those in the real-world.

Ask a crowd which hand I am holding a coin in – their wisdom might be split 50/50 in that case – no doubt this book would claim that they were only half a hand away with their group guess, which is better than 50% of the crowd’s guesses.

Maybe this book will dispel my initial fears with thorough and convincing analysis based on genuine reason. The reputation of renowned playwright Tom Stoppard depends on it.