What Big Data really brings to the table
The term ‘Big Data’ has become incredibly miss-used over a very short life. There are many ways in which it can be defined but, forget 3, 4 or 5 V’s for the moment and any other ways to categorise ‘Big Data’ and figure that instead of a type of usage of certain types of data, it’s more of a different way of thinking, it’s a new era in data processing if you like. Quite frankly the old tools based BI accompanied by the endless and often moronic discussions of Data Warehouses vs Data Marts etc was getting incredibly boring after 20 or so years. Lots of money was being spent, some wisely, some not and just a handful of vendors were making the whole market. Often as not, companies spent loads of money on Data Warehouse platforms because they were scared not to!
Now something has happened that is completely different. Do we have vast new sources of data all of a sudden? I doubt it. Are statisticians really the saviour of our planet? I doubt it. Do we need 900000 extra Data Scientists? I doubt it. Is statistics suddenly sexy? I think not. What has happened is that at last ther is a deep ‘feeling’ that analysing the data spawned from all sorts of sources is a valid and useful thing to do. At last, business people seem to have woken up to the facts that there is help out there to support or otherwise, there decisions and this is at the heart of Big Data. It’s a movement, a realisation that actually knowing something (OK, with a degree of certainty only) is far better that simple gut feeling. And off we go……..
Me? I’m excited. I wrote a book some time ago which pretty much outlined my disillusionment with BI over the last 20 years and now I’m excited again. Can we use data to impact and improve our daily lives as people (consumers) on this planet. I think we can but it won’t happen quickly and it won’t happen without a lot of discussions on privacy etc, etc.
So to be clear. For me Big Data represents a time in which we have suddenly (and it is sudden) become aware of the value in data. It’s not about a highly over-hyped Hadoop or any such technology, it’s about people, new opportunities and new realisations