De-batching and asynchronous processing of the messages

Question

Problem Description:

1.There is Biztalk Application receiving formatted/zipped data file containing > 2 million data records.

2.Created pipeline component that process file and 'de batching' these 2 million records of data into smaller slice-messages with ~2000 records each .

3.Slice-messages are being sent to SQL port and processed by stored procedure.Slice-messages contains filename and batch id.

Questions:
A.What would be the best way to know that all slice-messages received and processing of whole file completed on SQL side ?

B.Is there any way in biztalk port to say "do not send message of type B, until all messages of Type A have been send" (messages priority)?

Here are possible solutions I've tried :

S1.Add specific 'end of file' tags to end of last slice-message saying that file is being processed and stored procedure will receive this part of message mark file as completed. But because messages are being delivered asynchronously last message can be received on sql earlier that other messages and I will have false-competed event. So this solution is only possible for "Ordered delivery ports" - but this type of ports have poor performance because sending only one message at a time.

S2.Transfer total records count into every slice-message and run count() sql statement after every slice-message received. Because table where data is stored is super huge, even running count with filename as parameter takes time.

I'm wondering if there is better solution to know that all messages are being received ?

Answer 1

Have your pipeline component emit a "batch" message that contains the count of the records in the batch and some unique identifier that can link it back to the slice-messages records. Have both the stored procedure that processes the slice-message and the batch message check to see if the batch total (if it exists yet for the slice-message process) matches the processed total, if they match, then you've finished processing them all.

Answer 2

Here's how I would approach this.

1. Load the 2MM records into a SQL Server table or tables by SSIS.
2. Drain the table at whatever rate give you an acceptable performance profile.
3. Delete records as they are processed (completed).
4. When no more records for "FILE001.txt" exist, SQL Server will return a flag saying "FILE001.txt complete".
5. Do further processing.

When the staging table is empty, the Polling SP can either return nothing (the Adapter will silently ignore the response) or return a flag that says "nothing to do" and you handle that yourself.