SSIS Using Loop Container Through Report Ids to create separate csv files

Question

Let me preface this with saying I am still a newbie when it comes to SSIS. I am trying to replace a current MS Access process that has 13 queries that get created in excel and emailed to a list of users.

My SSIS package needs to get the report name, email dist list, report data, create a csv file and then send an email for each one.

I have created the following in SQL Server:

1. a table with the reports - each has a ReportId
2. a table with the users email addresses
3. a join table that links the user to each report
4. Created a stored proc when I pass in the ReportId and it returns the Email Dist List as well as some report details for use when I send my emails
5. Created a stored proc that I pass the ReportId to. In return I will get any of the values for that report. Each report has a different query that returns different columns of data.

I have also attempted to create an SSIS package that does the following:

1. using an Execute SQL Task get the list of Active Report Ids
2. a Foreach Container Loop set to ADO Enumerator to loop through each report Id
3. an Execute SQL Task that will return my report details and email dist list for each report id.

Then I tried placing a Data Flow Task inside of my loop and to run my stored proc to return the report data for each report id but I am getting an error message:

Error: SSIS Error Code DTS_E_OLEDBERROR.  An OLE DB error has occurred. 
Error code: 0x80040E55. An OLE DB record is available.  
Source: "Microsoft SQL Server Native Client 10.0"  Hresult: 0x80040E55  Description: "Column does not exist.". 

I am guessing this error is because my columns are not the same on each report. If that is the case, then do you have any suggestions about how to proceed?

What steps should I use in my SSIS package to go through each report? Do I need to create a separate SSIS package to run each report?

If I can run this in one package, do I need to create a flat file connection manager for each one of my csv reports that I generate?

Answer 1

I think your approach is sound, you just need to do something different on step 3. I think sp_send_dbmail might be an option (call it via Execute SQL Task).

You can assign your #6 query to the @query parameter (@query = "EXECUTE dbo.MakeReport @reportid=10') and @attach_query_result_as_file=1 and, assuming you've turned on database mail, it will send an email to the recipients with the results of your query as a CSV.

Use expressions to set the to, cc, and bcc based on the results of your first query and voila, spam-o-matic.

Answer 2

If your reports have different columns, you will need a different data flow for each of them. If you still want to do them in the same Foreach Loop, you can try doing something like having a variable like "reportType" that is populated when you run the Execute SQL Task. After that you can direct the flow there to the correct data flow. The following image shows an example of conditional flow:

In this tutorial, albeit having a different goal, the author describes how to create a conditional flow: http://www.simple-talk.com/sql/ssis/xml-configuration-files-in-sql-server-integration-services/

Here you will find more info about variables: http://sqlfool.com/2009/08/getting-started-with-variables-in-ssis/

Answer 3

Unfortunately, SSIS requires that the metadata of the data flow pipeline be set at design time, so if you are going to use a Data Flow task, then you will need a separate task for each report.

You can get around needing a separate Data Flow task and file connection manager for each report by not using a data flow. Instead you can use BCP (bulk copy program) to copy data to the file without needing to pre-define the metadata. BCP will take your stored procedure result set and just dump it out to the file. Dustin Ryan has a great article on how to do this on BIDN. There are certain cases where using the BCP in SSIS may be the better choice over a Data Flow Task if no transformations are needed, and this is one of those times.