Avoiding SSIS script task to convert utf-8 to unicode for AS400 data to SQL Server

Question

After many tries I have concluded that the optimal way to transfer with SSIS data from AS400 (non-unicode) to SQL Server is:

1. Use native transfer utility to dump data to tsv (tab delimited)

2. Convert files from utf-8 to unicode

3. Use bulk insert to put them into SQL Server

In #2 step I have found a ready made code that does this:

string from = @"\\appsrv02\c$\bg_f0101.tsv";
        string to = @"\\appsrv02\c$\bg_f0101.txt";
        using (StreamReader reader = new StreamReader(from, Encoding.UTF8, false, 1000000))
        using (StreamWriter writer = new StreamWriter(to, false, Encoding.Unicode, 1000000))
        {
            while (!reader.EndOfStream)
            {
                var line = reader.ReadLine();
                if (line.Length > 0)
                    writer.WriteLine(line);
            }
        }       

I need to fully understand what is happening here with the encoding and why this is necessary.

How can I replace this script task with a more elegant solution?

Answer 1

I don't have much insight into exactly why you need the utf-8 conversion task, except to say that SQL server - I believe - uses UCS-2 as its native storage format, and this is similiar to UTF-16 which is what your task converts the file to. I'm surprised SSIS can't work with a UTF-8 input source though.

My main point is to answer the "How could I replace this script task with a more elegant solution?":

I have had a lot of success using HiT OLEDB/400 Server. It allows you to set up your AS/400 / iSeries / System i / whatever IBM are calling it this week as a linked server in SQL server, and you can then access the 400's data directly from the server its linked to using the standard 4 part SQL syntax, e.g. SELECT * FROM my400.my400.myLib.myFile. Or even better, it's much more efficient as a passthrough query using EXEC...AT.

Using this you would not need SSIS at all, you'd just need a simple stored proc with that does an insert into your destination table direct from the 400 data.