Data warehouse - building an ETL template

Question

My company is trying to build a huge enterprise data warehouse and our company decided to hire a consulting firm to build it. How much time frame would it take to come up with an ETL Template? We are building a sql server 2012 data warehouse. We are in the phase of getting a project plan and we are very new to the data warehouse cycle. Any help will be appreciated.

Answer 1

The timeframe to come up with an ETL Template primarily depends on how experienced the Architects are. However, the approximate timeframe should be anywhere between 6 months and a year, depending on how complex your data is. DataStage and Ab Initio are some of the good ETL technologies that can be implemented for enterprise environments.

Consulting firms should first be able to estimate scalability of an ETL system across the lifetime of its usage. They have to be able to understand the volumes of data that must be processed within SLA that you mention. The time available to extract data changes from anywhere. Some ETL systems have to scale to process terabytes of data to update data warehouses with tens of terabytes of data. Increasing volumes of data may require designs that can scale from daily batch multiple day processing.

I would personally suggest you to inquire if the Consulting firm can implement parallel processing while trying to come up with a ETL solution because it will help you adapt to future requirements easily and at the same time speed up the ETL process.

Therefore, the factors that will be vital in defining a time frame for the template would be 1. Understanding the constraints & referential integrity of your Warehouse tables. 2. The amount of data that needs to be processed. 3. The various purposes that the ETL process will be used for.

Another main factor that you should primarily focus on is spend enough time in analyzing the requirements because you may not want to backtrack or change your approach to solutions frequently.

Good luck!

Answer 2

You need to consider a few things:

• Number of data sources
• Complexity of data sources
• Data quality
• Error handling and logging required
• Performance levels required

Also to influence the ETL model are the requirements around the complexity of the data warehouse itself, since a complex model will complicate the requirements on your ETL.

I also am of the opinion that it pays off to put some work upfront when designing your ETL framework, and to try to make it as "dynamic" as possible, as you don't want to be in a position of having to make changes to hundreds of ETL packages/processes towards the end of the project.

Without having much information I would say you would be looking at something between 2 weeks (for simpler projects) and 3 months (for complex ones).