Efficient way of writing to a text file in VB.NET

Question

We have some information that we need to write (about 18 KB) to a .txt file stored on one of our network drives. The file is re-written about once every 15 minutes, but it is read practically at least every second. We are currently using StreamWriter to write the file.

The file server is in a remote location and the round trip ping varies from less than 1 ms to 15 ms.

The problem is, sometimes it takes as long as six seconds to write the contents to the file, which is definitely way too long even after we take consideration of the network speed.

Therefore, I am just wondering if there is any efficient way to write the file using VB.NET to improve the performance? Java has a very good tool named BufferedOutputStream, which unfortunately is not available in VB.NET (or I just have not found it).

Answer 1

The fastest option:

Collect all your text into one large string first and then use System.IO.File.WriteAllText(text).

Answer 2

This function was written to read from a database and output to a text file. Please feel free to use it as a starting point.

 Sub MakeFile(ByVal Obj As Object)
    Dim CountRow As Integer = 0
    Dim TableName As String = CType(Obj, String())(0)
    Dim CommandText As String = CType(Obj, String())(1)


    Dim csvFileName As String = InitilizationSettings.DirectoryPath & TableName & ".txt"
    If File.Exists(csvFileName) Then
        File.Delete(csvFileName)
    End If
    Dim buffer() As Byte = {255}
    Dim FileObject As New FileStream(csvFileName, FileMode.OpenOrCreate)
    Dim MStream As New MemoryStream()
    Dim StreamWriterObj As New StreamWriter(MStream)
    Dim sb As New System.Text.StringBuilder
    Dim x As Integer = 0

    Dim reader As SqlDataReader
    Dim cmd As New SqlCommand()
    Dim conn As New SqlConnection(IOUtilities.GetConnectionString())


    conn.Open()


    With cmd
        .CommandText = CommandText
        .CommandTimeout = 1200
        .CommandType = CommandType.Text
        .Connection = conn
    End With

    reader = cmd.ExecuteReader()

    Do While reader.Read()
        'System.Console.Write("Loading rows to memory.../" & vbCr)
        sb.Append(Chr(34))
        sb.Append(reader.Item(0))
        'System.Console.Write("Loading rows to memory...|" & vbCr)
        sb.Append(Chr(34))
        For i = 1 To reader.FieldCount - 1
            sb.Append(",")
            sb.Append(Chr(34))
            sb.Append(reader.Item(i))
            sb.Append(Chr(34))
        Next
        'System.Console.Write("Loading rows to memory...\" & vbCr)



        sb.AppendLine()

        'Write every 10000 rows of data to the file from the buffer
        If x = 50000 Then
            StreamWriterObj.Write(sb.ToString().ToCharArray())
            MStream.Seek(0, SeekOrigin.Begin)
            MStream.WriteTo(FileObject)
            sb = New StringBuilder()
            CountRow = CountRow + x
            x = 0
        End If
        'System.Console.Write("Loading rows to memory...-" & vbCr)
        x = x + 1


        'LogEvents("Dumped " & strFileName & " to " & GetFilePath() & " at " & Now.ToString & vbCrLf & vbCrLf)
    Loop

    conn.Close()
    reader.Close()
    'Write any remaining data from the buffer to the file
    StreamWriterObj.Write(sb.ToString().ToCharArray())
    MStream.WriteTo(FileObject)
    FileObject.Close()

    System.Console.WriteLine(String.Format(vbCrLf & "Finished writing data to {1}", CountRow, csvFileName))

End Sub

Answer 3

Consider these in turn:

1. Create the file locally
2. Copy it to remote folder with a temporary extension
3. Rename the remote file to your original file name

Step 2 and 3 is as follows (using System.IO):

string OriginalExtension = ".ok", TemporaryExtension = ".dat";
string tmpFileRemote = RemoteFile.Replace(TemporaryExtension, OriginalExtension);
File.Copy(fileName, RemoteFile, true);
File.Copy(RemoteFile, tmpFileRemote, true);
File.Delete(RemoteFile);

The first File.Copy takes the time. But since it doesn't lock the real file people are using, it doesn't get locked. The second File.Copy actually just renames the file and replaces the real file with the one just uploaded. File.Delete deletes the temporary file uploaded.

Hope that helps.

Answer 4

There are a couple of things that come into effect here. But I've found that utilizing the IO.File.AppendText function speeds up the writing processes immensely.

Additionally, using the System.Text.Stringbuilder class as opposed as a traditional string (assuming you're concating the text to write to a file) improves the speed even more.

See Speeding up File Writing.

Answer 5

An old thread but I think there is still something to add:

Write all text at once using:
System.IO.File.WriteAllText(path As String, contents As String).

However it will still take time to write the file if it is being written over a slow connection to a remote location. To avoid having users read a partially written file you should write the data to a temporary file on the remote server. Once all the data has been written, copy the temporary file over the old file and then delete the temporary file.

If you want to be really sure no one reads a partially written file you can delete the old file and then move/rename the temporary file to replace the file you just deleted. In this case you must make sure the client programs gracefully handle file not found errors which will inevitably happen from time to time.

Answer 6

Consider writing to a local drive first, then moving that file to your network drive.

Answer 7

Sub writefile()
    Dim file As System.IO.StreamWriter
    file = My.Computer.FileSystem.OpenTextFileWriter("N:\GeneratedNumber.txt", False)
    file.WriteLine("Player1 Skill is " & Skill(0))
    file.WriteLine("Player1 Strength is " & Skill(1))
    file.WriteLine("Player2 Skill is " & Skill(2))
    file.WriteLine("Player2 Strength is " & Skill(3))
    file.Close()
End Sub