Competitions and the Wisdom of Crowds

John Robert Brown

A successful career in classical music eventually entails making decisions about the performance of others. Whether one conducts, plays, teaches, publishes, runs a festival, writes criticism or edits a magazine, if one is successful one is likely, sooner or later, to sit in judgement. In fact, we all do it. Even when choosing a CD or buying a concert ticket we assess another musician's performance.

The most prestigious type of evaluation is when one is invited to judge a major international competition. Normally in such events the judges are selected for their fame and achievements, not for their assessment skills. Membership of a distinguished panel is offered because of great performance, or successful teaching, not for understanding measures of central tendency, distributions, and sampling or simulation statistics.

But anyone who has spent any time in music education will be well aware of the fallibility of exams and examiners. That's why we appoint external examiners, why we double-mark student work, why we have inspection and appraisal and why we spend time teaching assessment techniques to teachers. Justice is all. One of the reasons why the juries at international music competitions are usually comprised of several members is to be seen to be fair. Competition juries vary in size. For example, at major piano competitions the adjudicating panel tends to be large; in recent years the Leeds International Pianoforte Competition had 14 jury members, 13 jurors adjudicated at Hamamatsu, there were 12 at the Van Cliburn competition in Fort Worth Texas and 10 at the Franz Liszt in Utrecht. Obviously, a large panel of judges, though sometimes unmanageable and inefficient, limits the impact of bias, whether through nationalism, nepotism or plain old narcolepsy. But a new publication argues that there is another, more powerful, reason for respecting the superiority of a large group over a small one in the matter of accurate judgement.

According to James Surowiecki's new book The Wisdom of Crowds, large groups of people are simply smarter than a few experts. To quote a basic example, two hundred students were asked to rank items by weight. The group's estimate was 94% accurate, better than all but five of the individual guesses. Many such experiments are cited by Surowiecki, where individual guesses were aggregated, then averaged. "The simplest way to get reliably good answers is just to ask the group each time," says Surowiecki. Not only is the judgement of crowds good in laboratory settings and classrooms, but this applies in the real world as well. Evidence is cited from gambling, the stock market, and even in the development of the algorithm used by the Internet search engine, Google.

Surowiecki cites evidence that when comparing the problem-solving skills of groups of people, a group made up of some smart people and some not-so-smart always did better than a group comprised of smart people only. He explains that smart groups that are too much alike find it harder to keep learning, because each member is bringing less and less new information to the panel. Of course, homogeneous groups are great at doing what they do well - playing the piano, in this case. But they become progressively less able to investigate alternatives. In The Wisdom of Crowds, Surowiecki quotes the organisational theorist James G. March: "Bringing new members into the organization, even if they're less experienced and less capable, actually makes the group smarter simply because what little the new members do know is not redundant with what everyone else knows." This gain does not come from the superior knowledge of the average new recruit. Recruits are, on average, less knowledgeable than the people they replace. The beneficial effect comes from their diversity.

Although Surowiecki considers committees, juries and teams - and even beauty contests - he neglects to consider the minefield that surrounds the judging of musical events. That's a pity. Surowiecki's book should be required reading for all music competition organisers. For example: two-thirds of all foremen are men, yet during deliberations they talk more than women do. Furthermore, did you realise that the order in which people speak has been shown to have an effect on the course of a discussion? Early comments are more influential. Though these points are hardly a shock, one wonders whether they are ever taken into account by administrators of music competitions.

What conclusions are here for such organisers? Well, my first advice is to read Surowiecki's book. A brief account here can't do it justice, but it is clear that juries should be large and diverse. Genders and ages should be mixed. Why should the jurors in a piano competition all be pianists? Why not violinists, or wind players? Even better, why not give the audience a vote? Eventually, they'll be voting by buying the concert tickets or the CDs. In conductors' competitions, or concerto rounds of piano competitions, why not give the orchestra an equal vote? Clearly, more participants would give a better, fairer, result. In the recent Van Cliburn competition (June 5th) an Internet prize was awarded to the pianist attracting the most votes via the competition's website.

Watching the jury during a competition is revealing, sometimes more than it should be. During the last Leeds Piano Competition I was quite shocked to see one member of the panel bury his head in his hands querulously over some perceived lapse by a contestant. Whether the unfortunate pianist noticed this rudeness, I can't say. But I do believe that such a gesture is another powerful argument for opening up the judgement to the wisdom of the crowd.

James Surowiecki. The Wisdom of Crowds. Abacus Books, £7.99. ISBN 0 349 11605 9

This article first appeared in Classical Music magazine, 18 June 2005. Used by permission.
Updated and maintained by: routeToWeb