Using SSIS to read flat files with multiple record types

Question

We are evaluating SSIS to see if it will be appropriate for a new project that is coming up. One of the processes will have to process a flat file with delimited records. The file will contain orders. There is a header line, an (optional) shipping address line, and one or more detail lines. Each line's fields are delimited but are not the same format.

I read this answer:

SSIS transactional data (different record types, one file)

And I can split the data using the Conditional Split task to produce several outputs, but am not sure how to proceed from there. I have two issues that I need to resolve:

1. The order header should be inserted first, before the address and details since the address and details will reference the order record, so I think I need to process that output first, but I'm not sure in SSIS how to make that branch of the Conditional Split task be processed before the other branches. Ideally, I would like to process the order header and then store the order id in a user variable so that when processing the details, I can reference that variable.

2. There will be multiple orders in the file, so splitting it is more complex.

I could always write an application in C# that will preprocess the file or read the file into a staging table, but I'm not sure I like that approach.

Can anyone who has been through this process share some insights into how they dealt with it?

Thanks,

Chris

Answer 1

After the split, deposit each type of record into it's own staging table - or into a ssis raw data destination, which is faster and good for intermediate steps like this. Then load all the headers into their final table and proceed without referential errors.

I'm assuming the detail records have a headerID in them? That should make dealing with your 2nd question easy. If not, let us know.