In the 18th century, scientists started to believe that everything was mechanistic, that everything that happened was governed by rules. All you had to do was understand the rules and you would know what was going to happen.
Pierre Simon Laplace, a French Marquis and polymath, wrote a book explaining how through observation and computation, you could predict the future position of the planets and comets. Admittedly, his formulae contained a “fudge factor” to account for small differences in the predicted position and the actual position of these celestial bodies.
It was thought that these fudge factors were necessary to make up for the inaccuracy of the initial measurements and if you could improve the precision of the input data, the output positions would naturally become more accurate.
Unfortunately, this was not the case. The more precise the measurements, the more errors were found in the predicted positions. This was true of most areas of science; improving the measurements did not mean that the predicted outcomes would become closer to reality. The deterministic universe was seriously wounded.
A new realisation swept science that the universe was (and is) much more probabilistic. So a new science was born, statistics. As with all scientific revolutions, it took a great deal of time for people to change their minds and overcome their prejudices about something as certain as science being dependent on probabilities.
But despite initial scepticism, the scientific community was won over. Universities added statistics courses to the mathematics syllabus and even opened separate departments to teach the new science.
Over time, it was realised that this new science was not just a case of conducting an experiment or carrying out a survey, then tabulating the data and voila, you had the answer. What became clear was that working with statistics required discipline. Collecting data was not enough. Studies needed to be carefully structured in order to ensure the data was relevant and accurate, in order not to build in bias and truly answer the question that was being asked. For example, medical researchers discovered that the human mind, both of the participant and the person carrying out the trial, could impact the outcome of a drug trial. Double-blind studies became the norm.
Today experiments and surveys are carefully designed and data are collected and manipulated, with results expressed in orders of confidence. The design of the experiment and the understanding of how to analyse the results require highly skilled people.
Surveys in which people are asked questions are particularly susceptible to bias. As scientists like Kahneman and Tversky have shown in studies of decision making, humans are not rational and easily primed to make certain responses or reflect biases.
It is a universal truth that the output of research can be only as good as the quality of data being analysed. Never better and often worse.
You may remember that the UK Gambling Commission (GC) used to rely on the National Health Service’s annual survey to gather information about gambling behaviours in England. This is a face-to-face survey primarily about health that included some questions about gambling.
Whilst the survey was useful, it did not include the questions that the Department for Culture and Sport needed in order to develop policy or the GC to design regulations and see the impact of those regulations. It was unwieldy, costly, and covered only England, not the rest of Great Britain.
The GC decided to design and implement a new, fully comprehensive, online survey to give it the data that it and its sponsoring ministry would need to inform policy and see the impact of any changes that were made to regulations.
What seemed like a very good idea, however, became mired in controversy. Far from giving the GC what it needed, the survey has produced results that are from a sample that is not representative of the whole population (online surveys attract those who want to participate in those particular surveys). The responses were, I believe, inadvertently primed to overstate rates of problem gambling and gambling harms and because of these shortcomings, there is a high probability that they are irrelevant and nearly useless.
Earlier this year, the results of an independent assessment of the survey were announced by the GC under the headline “Independent assessment endorses Gambling Survey for Great Britain.” The GC stated that the survey assessor had found the survey to be “exemplary in all respects”. The assessor had not found that. In fact, the assessor, Professor Patrick Sturgis, wrote, “There is a non-negligible risk that they [the survey’s estimates] substantially overstate the true level of gambling and gambling harm in the population”.
Once the details of the survey and the methodology became public and people had a chance to understand its considerable shortcomings, pressure was brought to bear on the regulator by some parts of the industry.
In a climbdown, prior to the release of the second “wave” of the survey findings in July, the GC recognised the survey’s weaknesses and forewarned the news media and public that the survey findings should not be compared to data from previous NHS surveys to determine trends or extrapolated to indicate levels of problem gaming, gambling harms, etc., for the entire population. Not surprisingly, some in the media ignored the warning.
The idea of a new survey was undoubtedly a good one, but large-scale surveys of this nature take considerable time to develop and refine. The survey data was released before the survey itself was fully cooked.
Why the GC decided to move ahead and publish its survey, given that the results were not very useful, will probably never be known publicly. It is certainly possible that having decided to go down this path and wanting something to use as a baseline to see the impact of new regulations and having announced that it was an “exemplary survey in all respects”, they could not make a U-turn and had to keep to the published timeline. Whatever the reason, what is certain is that the GC’s reputation has been badly tarnished by this unfortunate episode.
The GC now seems to be indicating that the survey will need to be “tweaked” to allow it to produce more useful data. As Melanie Ellis, a partner at Northbridge Law, pointed out, if the survey is changed, it can no longer be considered a baseline and “we will have no reliable way of assessing whether rates of gambling-related harm have reduced since before their introduction.”
By publishing the results of the survey prematurely, the Commission has wasted an exceptional opportunity.
