Note, very soon I’m going to take a diversion in this blog to discuss Big Data………………so bear with me whilst I sort my feelings out!!
Firstly, it’s only right to say that Data Marts and Data Warehouses have often brought great benefits to those companies that have deployed them well. Some companies have been smart and gained real benefits quickly, but others have simply got bogged down in a morass of complexity and politics and simply wasted huge amounts of time and money. Failure comes in many flavours and it won’t do anyone any good for me to list all the pitfalls here, so when I talk about ‘problems’, I want to be specific and remain relevant to the theme of this blog
If we examine the biggest problem that confronts BI, it is the fact that the base data of a company is invariably spread across many systems, and is subject to many movements and relocations, subject to many modifications in different systems, and subject to very little validation.
The greatest driver in IT architecture must therefore surely be to remove all of this complexity, and drive to a point where there is no copy management of data because it is simply not needed, and it’s this principle that convinces me that our vision of the Data Warehouse must change. Today, in order to solve the problems caused by inaccurate data, we actually add a further requirement for copy management as data enters the Data Warehouse. In effect we take lots of unorganised data and make a further copy of it, change it, and summarise it until it often becomes unrecognisable and certainly irreconcilable back to its original.
Now, there are no clear alternatives to what I have outlined above, but it does seem incredible that in the 21st century we are still solving problems caused by one flawed strategy by adding a further requirement for that same flawed strategy.