How to write stored procedure output directly to a file on an FTP without using local or temp files?

Question

I want to get the results of a stored procedure and place them into a CSV file onto an FTP location.

The catch though is that I cannot create a local/temporary file that I can then FTP over.

The approach I was taking was to use an SSIS package to create a temporary file and then have a FTP Task within the pack to FTP the file over, but our DBA's do not allow temporary files to be created on any servers.

in reply to Yaakov Ellis

I think we will need to convince the DBA's to let me use at least a share on a server that they do not operate, or ask them how they would do it.

in reply to Kev

I like the idea of the CLR integration, but I don't think our DBA's even know what that is lol and they would probably not allow it either. But I will probably be able to do this within a Script Task in an SSIS package that can be scheduled.

Answer 1

This step-by-step example is for others who might stumble upon this question. This example uses Windows Server 2008 R2 server and SSIS 2008 R2. Even though, the example uses SSIS 2008 R2, the logic used is applicable to SSIS 2005 as well. Thanks to @Kev for the FTPWebRequest code.

Create an SSIS package (Steps to create an SSIS package). I have named the package in the format YYYYMMDD_hhmm in the beginning followed by SO stands for Stack Overflow, followed by the SO question id, and finally a description. I am not saying that you should name your package like this. This is for me to easily refer this back later. Note that I also have two Data Sources namely Adventure Works and Practice DB. I will be using Adventure Works data source, which points to AdventureWorks database downloaded from this link. Refer screenshot #1 at the bottom of the answer.

In the AdventureWorks database, create a stored procedure named dbo.GetCurrency using the below given script.

CREATE PROCEDURE [dbo].[GetCurrency]
AS
BEGIN
    SET NOCOUNT ON;
    SELECT 
    TOP 10      CurrencyCode
            ,   Name
            ,   ModifiedDate 
    FROM        Sales.Currency
    ORDER BY    CurrencyCode
END
GO

On the package’s Connection Manager section, right-click and select New Connection From Data Source. On the Select Data Source dialog, select Adventure Works and click OK. You should now see the Adventure Works data source under the Connection Managers section. Refer screenshot #2, #3 and #4.

On the package, create the following variables. Refer screenshot #5.

• ColumnDelimiter: This variable is of type String. This will be used to separate the column data when it is written to the file. In this example, we will be using comma (,) and the code is written to handle only displayable characters. For non-displayable characters like tab (\t), you might need to change the code used in this example accordingly.

• FileName: This variable is of type String. It will contain the name of the file. In this example, I have named the file as Currencies.csv because I am going to export list of currency names.

• FTPPassword: This variable is of type String. This will contain the password to the FTP website. Ideally, the package should be encrypted to hide sensitive information.

• FTPRemotePath: This variable is of type String. This will contain the FTP folder path to which the file should be uploaded to. For example if the complete FTP URI is ftp://myFTPSite.com/ssis/samples/uploads, then the RemotePath would be /ssis/samples/uploads.

• FTPServerName: This variable is of type String. This will contain the FTP site root URI. For example if the complete FTP URI is ftp://myFTPSite.com/ssis/samples/uploads, then the FTPServerName would contain ftp://myFTPSite.com. You can combine FTPRemotePath with this variable and have a single variable. It is up to your preference.

• FTPUserName:This variable is of type String. This will contain the user name that will be used to connect to the FTP website.

• ListOfCurrencies: This variable is of type Object. This will contain the result set from the stored procedure and it will be looped through in the Script Task.

• ShowHeader: This variable is of type Boolean. This will contain values true/false. True indicates that the first row in the file will contain Column names and False indicates that the first row will not contain Column names.

• SQLGetData: This variable is of type String. This will contain the Stored Procedure execution statement. This example uses the value EXEC dbo.GetCurrency

On the package’s Control Flow tab, place an Execute SQL Task and name it as Get Data. Double-click on the Execute SQL Task to bring the Execute SQL Task Editor. On the General section of the Execute SQL Task Editor, set the ResultSet to Full result set, the Connection to Adventure Works, the SQLSourceType to Variable and the SourceVariable to User::SQLGetData. On the Result Set section, click Add button. Set the Result Name to 0, this indicates the index and the Variable to User::ListOfCurrencies. The output of the stored procedure will be saved to this object variable. Click OK. Refer screenshot #6 and #7.

On the package’s Control Flow tab, place a Script Task below the Execute SQL Task and name it as Save to FTP. Double-click on the Script Task to bring the Script Task Editor. On the Script section, click the Edit Script… button. Refer screenshot #8. This will bring up the Visual Studio Tools for Applications (VSTA) editor. Replace the code within the class ScriptMain in the editor with the code given below. Also, make sure that you add the using statements to the namespaces System.Data.OleDb, System.IO, System.Net, System.Text. Refer screenshot #9 that highlights the code changes. Close the VSTA editor and click Ok to close the Script Task Editor. Script code takes the object variable ListOfCurrencies and stores it into a DataTable with the help of OleDbDataAdapter because we are using OleDb connection. The code then loops through each row and if the variable ShowHeader is set to true, the code will include the Column names in the first row written to the file. The result is stored in a stringbuilder variable. After the string builder variable is populated with all the data, the code creates an FTPWebRequest object and connects to the FTP Uri by combining the variables FTPServerName, FTPRemotePath and FileName using the credentials provided in the variables FTPUserName and FTPPassword. Then the full string builder variable contents are written to the file. The method WriteRowData is created to loop through columns and provide the column names or data information based on the parameters passed.

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.Data.OleDb;
using System.IO;
using System.Net;
using System.Text;

namespace ST_7033c2fc30234dae8086558a88a897dd.csproj
{
    [System.AddIn.AddIn("ScriptMain", Version = "1.0", Publisher = "", Description = "")]
    public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
    {

        #region VSTA generated code
        enum ScriptResults
        {
            Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
            Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
        };
        #endregion

        public void Main()
        {
            Variables varCollection = null;

            Dts.VariableDispenser.LockForRead("User::ColumnDelimiter");
            Dts.VariableDispenser.LockForRead("User::FileName");
            Dts.VariableDispenser.LockForRead("User::FTPPassword");
            Dts.VariableDispenser.LockForRead("User::FTPRemotePath");
            Dts.VariableDispenser.LockForRead("User::FTPServerName");
            Dts.VariableDispenser.LockForRead("User::FTPUserName");
            Dts.VariableDispenser.LockForRead("User::ListOfCurrencies");
            Dts.VariableDispenser.LockForRead("User::ShowHeader");
            Dts.VariableDispenser.GetVariables(ref varCollection);

            OleDbDataAdapter dataAdapter = new OleDbDataAdapter();
            DataTable currencies = new DataTable();
            dataAdapter.Fill(currencies, varCollection["User::ListOfCurrencies"].Value);

            bool showHeader = Convert.ToBoolean(varCollection["User::ShowHeader"].Value);
            int rowCounter = 0;
            string columnDelimiter = varCollection["User::ColumnDelimiter"].Value.ToString();
            StringBuilder sb = new StringBuilder();
            foreach (DataRow row in currencies.Rows)
            {
                rowCounter++;
                if (rowCounter == 1 && showHeader)
                {
                    WriteRowData(currencies, row, columnDelimiter, true, ref sb);
                }

                WriteRowData(currencies, row, columnDelimiter, false, ref sb);
            }

            string ftpUri = string.Concat(varCollection["User::FTPServerName"].Value,
                                          varCollection["User::FTPRemotePath"].Value,
                                          varCollection["User::FileName"].Value);

            FtpWebRequest ftp = (FtpWebRequest)FtpWebRequest.Create(ftpUri);
            ftp.Method = WebRequestMethods.Ftp.UploadFile;
            string ftpUserName = varCollection["User::FTPUserName"].Value.ToString();
            string ftpPassword = varCollection["User::FTPPassword"].Value.ToString();
            ftp.Credentials = new System.Net.NetworkCredential(ftpUserName, ftpPassword);

            using (StreamWriter sw = new StreamWriter(ftp.GetRequestStream()))
            {
                sw.WriteLine(sb.ToString());
                sw.Flush();
            }

            Dts.TaskResult = (int)ScriptResults.Success;
        }

        public void WriteRowData(DataTable currencies, DataRow row, string columnDelimiter, bool isHeader, ref StringBuilder sb)
        {
            int counter = 0;
            foreach (DataColumn column in currencies.Columns)
            {
                counter++;

                if (isHeader)
                {
                    sb.Append(column.ColumnName);
                }
                else
                {
                    sb.Append(row[column].ToString());
                }

                if (counter != currencies.Columns.Count)
                {
                    sb.Append(columnDelimiter);
                }
            }
            sb.Append(System.Environment.NewLine);
        }
    }
}

Once the tasks have been configured, the package’s Control Flow should look like as shown in screenshot #10.

Screenshot #11 shows the output of the stored procedure execution statement EXEC dbo.GetCurrency.

Execute the package. Screenshot #12 shows successful execution of the package.

Using the FireFTP add-on available in FireFox browser, I logged into the FTP website and verified that the file has been successfully uploaded to the FTP website. Refer screenshot #13.

Examining the contents by opening the file in Notepad++ shows that it matches with the stored procedure output. Refer screenshot #14.

Thus, the example demonstrated how to write results from database to an FTP website without having to use temporary/local files.

Hope that helps someone.

Screenshots:

#1: Solution_Explorer

#2: New_Connection_From_Data_Source

#3: Select_Data_Source

#4: Connection_Managers

#5: Variables

#6: Execute_SQL_Task_Editor_General

#7: Execute_SQL_Task_Editor_Result_Set

#8: Script_Task_Editor

#9: Script_Task_VSTA_Code

#10: Control_Flow_Tab

#11: Query_Results

#12: Package_Execution_Successful

#13: File_In_FTP

#14: File_Contents

Answer 2

If you were allowed to implement CLR integration assemblies you could actually use FTP without having to write a temporary file:

public static void DoQueryAndUploadFile(string uri, string username, string password, string filename)
{
    FtpWebRequest ftp = (FtpWebRequest)FtpWebRequest.Create(uri + "/" + filename);
    ftp.Method = WebRequestMethods.Ftp.UploadFile;
    ftp.Credentials = new System.Net.NetworkCredential(username, password);

    using(StreamWriter sw = new StreamWriter(ftp.GetRequestStream()))
    {
        // Do the query here then write to the ftp stream by iterating DataReader or other resultset, following code is just to demo concept:
        for (int i = 0; i < 100; i++)
        {
            sw.WriteLine("{0},row-{1},data-{2}", i, i, i);
        }
        sw.Flush();
    }
}

Answer 3

Is there a server anywhere that you can use where you can create a temporary file? If so, make a web service that returns an array containing the contents of the file. Call the web service from the computer where you can create a temporary file, use the contents of the array to build the temp file and ftp it over.

If there is no where at all where you can create a temporary file, I don't see how you will be able to send anything by FTP.

Answer 4

Try using a CLR stored procedure. You might be able to come up with something, but without first creating a temporary file, it might still be difficult. Could you set up a share on another machine and write to that, and then ftp from there?

Answer 5

Script from the FTP server, and just call the stored proc.

Answer 6

The catch though is that I cannot create a local/temporary file that I can then FTP over.

This restriction does not make any sense, try to talk to DBA nicely and explain it to him/her. It is totally reasonable for any Windows process or job to create temporary file(s) in appropriate location, i.e. %TEMP% folder. Actually, SSIS runtime itself often creates temporary files there - so if DBA allows you to run SSIS, he is allowing you to create temporary files :).

As long as DBA understands that these temporary files do not create problem or additional workload for him (explain that he does not have to set special permissions, or back them up, etc), he should agree to let you create them.

The only maintenance task for DBA is to periodically clean %TEMP% directory in case your SSIS job fails and leaves the file behind. But he should do this anyway, as many other processes may do the same. A simple SQL Agent job will do this.