BIG Data -Problems with Contemporary Solutions


Note, very soon I’m going to take a diversion in this blog to discuss Big Data………………so bear with me whilst I sort my feelings out!!

Firstly, it’s only right to say that Data Marts and Data Warehouses have often brought great benefits to those companies that have deployed them well. Some companies have been smart and gained real benefits quickly, but others have simply got bogged down in a morass of complexity and politics and simply wasted huge amounts of time and money. Failure comes in many flavours and it won’t do anyone any good for me to list all the pitfalls here, so when I talk about ‘problems’, I want to be specific and remain relevant to the theme of this blog

If we examine the biggest problem that confronts BI, it is the fact that the base data of a company is invariably spread across many systems, and is subject to many movements and relocations, subject to many modifications in different systems, and subject to very little validation.

The greatest driver in IT architecture must therefore surely be to remove all of this complexity, and drive to a point where there is no copy management of data because it is simply not needed, and it’s this principle that convinces me that our vision of the Data Warehouse must change. Today, in order to solve the problems caused by inaccurate data, we actually add a further requirement for copy management as data enters the Data Warehouse. In effect we take lots of unorganised data and make a further copy of it, change it, and summarise it until it often becomes unrecognisable and certainly irreconcilable back to its original.

Now, there are no clear alternatives to what I have outlined above, but it does seem incredible that in the 21st century we are still solving problems caused by one flawed strategy by adding a further requirement for that same flawed strategy.



About bibongo

I'm a consultant in the field of Business Intelligence and have been since the mid 80's which gives you some idea of my age! I'm priviledged to have held senior positions with Teradata, Oracle, Hp and EMC. I have an English son and a Swedish daughter seperated by some 18 years which is another type of welcome challenge!
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s