Big data is failing. That’s right!
Anyone who hasn’t been living under a rock is aware of how essential mass data gathering has become around the world. Not only are major political campaigns dependent on huge polling drives to keep track of who’s ahead, but private citizens’ data is gathered to better inform politicians on how to behave and what to endorse.
But with all this data harvesting going on, and all the heavy emphasis on how beneficial the use of analytics can be in the public and private sectors, why is it that over the last couple of years major polling agencies in some of the biggest political contests have been unable to make even vaguely accurate predictions? And individual commentators/journalists like Nicco Mele have? (See his article: 7 Reasons Why Trump Will Win.)
What’s going on?
There was a time back in the 60’s that eminent philosophers and sci-fi authors believed that by analyzing the data gathered on large populations of people, you would one day not only be able to make accurate assumptions about the results of elections, but you would literally be able to predict the future of human history. The theory was called Pychohistory. This seems to not only have been disproven lately but to an embarrassing degree.
Polling Data and the British…
In the UK, Brexit was the second major event to be utterly misinterpreted by the analysts. In 2015 the pollsters predicted a neck-and-neck race between the Conservative and Labor parties, but the result was a decimation of Ed Milliband’s Labor Party. In America, once again, we have seen pollsters broadly agreeing that Hillary Clinton would be a shoo-in for the top job, but we now know that Donald J. Trump had a comfortable lead all along. So why is all our data WRONG?!?!?
The Big Issue
Google, among others, have been so enthusiastic about data gathering that they are happy to glaze over the fact that they still can’t tell when someone else is using your phone, or whether a click is deliberate or accidental. But their biggest problem is that, no matter how much data one gathers, it will only reveal what people want to tell you, not what they feel. There are some ways of improving this, including more layered surveying, but at present most opinion polls are carried out in a simple, tick-the-box fashion. It’s just harder to gather data any other way.
In the world of politics, this can be dicey, as all a politician really wants to know is what you will actually do in the voting booth, and this problem flows into the private sector too. There have been countless times that businesses big and small have tested the water with their customers before making a change, been happy with the results, and seen a product change fail anyway. (The new Toblerone scandal comes to mind). So what can data analysts plugged into Google Analytics and fancy heatmaps take away from this? And why is it that the bookies are better at predicting election results that the pollsters?
The Human Touch
The key is to appreciate that although there have been drastic advances in data gathering techniques, and things are likely to progress at a steady clip, it is important not to blindly follow the numbers. That’s why having experienced analysts draw conclusions in coordination with reliable hard data is necessary. Many analytics companies will drown you in the world of Big Data, but only a few will talk to you, and offer advice on the data you receive — these companies are valuable.
Qualitative data is still important, even if it’s going out of fashion.
