Flexible way to implement analytical data warehouse with changing attribute structure?

Question

I am currently searching alternative ways to implement management of source data and results from analytical processess. As anyone can imagine we have dozens of different OLTP-databases based on handling some specific business process and few ERP's too. They have been developed different times during the operations of our companuy and content and application architechture from database point of view differ a lot. And they have different solutions for storage of historical information and change tracking for attributes.

Many in our organization do not like idea about DW-solution because they have bad experiences from development of datamarts for reporting or other related projects.

So for that reason many have suggested that we should just store data in file-system in binary file formats or in csv or xml files. And augment that data with direct access to OLTP's via ODBC/JDBC interface.

I am concerned that this approach cannot anyway guarantee integrity and point in time correctness of data. Data could be too volatile and contain errors which are not always be possible to correct when running analytics.

And we do not have a place to store scoring results yet. There is a suggestion that these could be written to flat-file and exported to the target system to use in there (like in CRM for campaign-management).

Is there some way I could rationalize a need to more formal data-management? I am thinking that we need a EDW solution.

Regards,

Analyst

Answer 1

sounds like your environment might be a good candidate for the data-vault approach. Check out articles by Daniel Linstedt and if you want a good highlevel overview you can purchase the following book: Super Charge your Data Warehouse.