Using the "using" statement with StreamWriter/Reader inside loop

Question

Here is my situation.

1. Read one line from a text file
2. "Process" the line
3. Write the "processed" line to a new text file
4. Loop to #1 and repeat until EOF

Here is how I'm doing it:

using (StreamReader srReader = new StreamReader(strInputFile))
{
    // loop until EOF
    while ((strCurrentLine = srReader.ReadLine()) != null)
    {
        // "process" strCurrentLine...

        // write the "processed" strCurrentLine to a new text file
        using (StreamWriter sw = new StreamWriter(strFullOutputPathFileName, false))
        {
            // write strCurrentLine to the new file
            sw.WriteLine("stuff...");
        }
     }
}

My manager tells me that using the using statement like I am inside of a loop will extremely hinder performance. The reason is because the StreamWriter instance will be created as many times as I'm looping. So, if I looped 1,000 times, I'd have 1,000 instances of my StreamWriter object, which could severely hinder performance.

Is this true? Also, is my method above the best way to accomplish this?

Answer 1

My manager tells me that using the using statement like I am inside of a loop will extremely hinder performance. The reason is because the StreamWriter instance will be created as many times as I'm looping. So, if I looped 1,000 times, I'd have 1,000 instances of my StreamWriter object, which could severely hinder performance.

Is this true?

Well, it's true, but not because you're creating instances, but because you're opening and closing a file 1,000 times. You could create 1,000 strings with almost no impact to performance.

Also, is my method above the best way to accomplish this?

To start, move the writer creation outside of the while loop:

using (StreamReader srReader = new StreamReader(strInputFile))
{
    // write the "processed" strCurrentLine to a new text file
    using (StreamWriter sw = new StreamWriter(strFullOutputPathFileName, false))
    {
        // loop until EOF
        while ((strCurrentLine = srReader.ReadLine()) != null)
        {
            // "process" strCurrentLine...

            // write strCurrentLine to the new file
            sw.WriteLine("stuff...");
        }
     }
}

However, you could also read the entire file into memory, process it, and write it out in one operation. The impact will depend on the processing that's done and whether you want partial results if there's an error.

Answer 2

Change your code to:

using (StreamReader srReader = new StreamReader(strInputFile))
{
    using (StreamWriter sw = new StreamWriter(strFullOutputPathFileName, false))
    {
        while ((strCurrentLine = srReader.ReadLine()) != null)
        {
            sw.WriteLine("stuff...");
        }
    }
}

And also, check Jon's comment about appending.

Answer 3

If your file isn't big you can use File.ReadAllLines to get file lines in a string array and do your processing afterwards.

Answer 4

Yes, you will be creating many StreamWriters by doing it that way (not many instances running simultaneously, however). A simple way to solve this is by creating the StreamWriter before you even enter the while loop.

Answer 5

You should open and close the file as few times as possible. So open it (create streamwriter) before the while loop. In the loop use sw.WriteLine("stuff...") After the loop call sw.Close().

Your manager is wrong in sense that it won't create 1000 instances because each instance will be released at the end of the iteration, but you'll be opening and closing the file which will impact the performance.

Regards, Damjan