Loading last 7 days files with specific names in SSIS

Question

I have a folder which gets a daily dump of 7 or 8 .xls files. Among all those files, I just need to load 2 files with a specific name.. ABC_datestamp.xls and XYZ_datestamp.xls.

The problem is I am not able to isolate these 2 file names because there are so many other files in the same folder. The second challenge is that my manager wants me to load all files from last week at once, which means I have to specifically load files from last 7 days only.

Right now I am copying those specific 7 files in to a folder and then loading it using a for each loop container.

Is there anyway, I can use a script task to generate the file names like ABC_02-Dec-2011.xls, ABC_01-Dec-2011.xls, ABC_30-Nov-2011.xls .......last 7 days files and pass it as a variable to a DATA FLOW TASK?

Any better suggestions ?

I know a lot of people have asked similar questions but I am new to SSIS and not able to find good answers.

Thanks a lot..

Answer 1

Rather than try and build a list of all possible file names, I'd loop through all the files in the folder and use a Script Task to determine which files to process further.

The control flow would look like this:

with FELC_InputFolderContents's variable mappings set thusly:

SCR_SetProcessFileFlag's configuration:

SCR_SetProcessFileFlag's script like so:

    public void Main()
    {
        string targetFileFullPath = (string)Dts.Variables["User::TargetFileFullPath"].Value;
        string targetFileName = Path.GetFileName(targetFileFullPath);
        Dts.Variables["ProcessFileFlag"].Value = IsValidFileName(targetFileName);
        Dts.TaskResult = (int)ScriptResults.Success;
    }

    private bool IsValidFileName(string targetFileName)
    {
        bool result = false;
        if (Path.GetExtension(targetFileName) == "xls")
        {
            string fileNameOnly = Path.GetFileNameWithoutExtension(targetFileName);
            if (fileNameOnly.Substring(0, 4) == "ABC_" || fileNameOnly.Substring(0, 4) == "XYZ_")
            {
                string datePart = fileNameOnly.Substring(4);
                DateTime dateRangeStart = DateTime.Today.AddDays(-7);
                DateTime dateRangeEnd = DateTime.Today;
                DateTime fileDate;
                if (DateTime.TryParse(datePart, out fileDate)) 
                {
                    if (dateRangeStart <= fileDate && fileDate <= dateRangeEnd)
                    {
                        result = true;
                    }
                }
            }
        }
        return result;
    }

and the precedence constraint between SCR_SetProcessFileFlag and DTT_ProcessSelectedFile configured thusly:

(Obviously, the specific logic for IsValidFileName depends on your particular requirements.)

Answer 2

Example in the following link explains how to loop through files in a folder and archive files that match a given month. In the example, the files are named as myfile 20-08-2011, myfile 13-09-2011, based on the month value, those files are then moved to an archive folder with month names like August and September etc.

The example simply archives the files. If you want to load data, then you need to place a Data Flow Task. within the Foreach loop container.

If you go through the example, you will get an idea to resolve your problem.

How do I create a package that would copy all files from a given folder into a new folder?

Here is an example that shows how to loop through a folder and load Excel files data into a database.

How to import Excel files with different names and same schema into database?