How do I create a database that receives data from other DBs, manual entry and data exports to create a web dashboard?

Question

I need to create a database which allow for easy manual entry of data as well as linking to SQL databases I have access to. This is something I've been researching and trying to design by myself for a while now. Everytime I think I have a solution I begin to have doubts that this is the right direction. I would really REALLY appreciate your help/input.

My current workflow and nightmare.
I work in IT and I'm responsible for reporting for my own group but it has been and will continue to expand across many groups across the whole IT organization. Each group has their own reporting tool with some that can export data cleanly/consistently, others that have to input data manually and a few which I can access the SQL database directly but haven't yet.

To create a monthly report I have a master workbook with graphs that reference data in multiple data collection workbooks. In these data collection workbooks I have the raw data which either myself or the team/service owners inputs each month. From that raw data I may create a pivot table to get the data I want and then in each one of the collection workbooks I have a 'reference" tab. In this reference tab I have a formula that is a rolling 13 months in the first column which say goes from Jan 2013 to Jan 2014. The months are calculated by referencing a cell that has the current reporting month in it. So for this example its Jan 2014. When you change that cell to the next month say Feb 2014 it will roll that first column so that those same cells now go from Feb 2013 to Jan 2014. the columns to the right of this column reference either the data table directly or the pivot tables using vlookups() or getpivotdata() formulas that reference the month cell to the left and grabs data accordingly. Here is an example of a collection workbook.

As you can see this can be a very manual and tedious workflow with many points where this could be improved. One major issue is the fact that each team/group doesn't pull data from the same tool and doesn't serve it up in the right format. I sometimes have to write a vba script to take the raw data and add helper data like a column with the correctly formatted 02/01/2014 format so it works in pivot tables correctly instead of multiple entries with different days (ie. 02/05/2014 02/23/2014).

The end goal + where I want to be
I want to create a database where I can link SQL databases directly to it as well as making it easy for a team to manually input data by reading in a csv or xls file of data. The crux again is that I'm only one person and I want to offload as much as the data entry to each one of those teams/groups without making it a difficult task.

Using the master sheet to create a monthly report has some great results but I would like this to be a web accessible and dynamic dashboard. I'm a pretty good front end dev with knowledge of HTML, CSS+LESS and some JS+SQL knowledge. I will have support of a senior web developer if I can proceed with this project. I just need help in working out the data collection/storage/querying portion of this.

If you got this far thank you so much for reading about my problem. If I wasn't clear about something or you have more questions for me please feel free to ask.

THANKS!

Answer 1

You may want to take a look at SQL Server's linked servers, according to the documentation, you can link any database who have an OLE DB driver so you can run queries mixing SQL Server and other databases, so it may be worth your while to check this.