Performance issues creating an XML file from Database

Question

I'm trying to write an XML file, which is a Google Product feed, but it's very slow.

Having done quite a bit of Googling, I am not sure of the fastest approach, except for breaking it into smaller files.

I have about 3500 products currently, and this is what I am doing.

Dim fs As New FileStream(HttpContext.Current.Server.MapPath("LOCATION OF FEED XML"), FileMode.Create)
Dim writer As New XmlTextWriter(fs, System.Text.Encoding.UTF8)

writer.WriteStartDocument()
   ' ==== LOAD DATABASE INTO OdbcDataReader CALLED 'd' ==== '
While d.Read
   try
        writer.WriteStartElement("item")

        writer.WriteStartElement("g:id")
        writer.WriteString(pID)
        writer.WriteEndElement()
        '==== etc 
    catch
        ignore broken ones
    end
End While

     writer.WriteEndElement()
     writer.WriteEndDocument()
     writer.Close()

 d.Dispose()
 connection.Dispose()
 connection.Close()

I am not familiar with FileStream, MemoryStream etc, so not really sure what is exactly happening here where the writing is concerned.

Presumably writing to memory would be faster, but what's the process to do that and then save to disk?

Would the amount of data be an issue for memory?

Obviously I am coding with VB, using .NET 2.0

Any suggestions appreciated.

Answer 1

The performance problem doesn't seem to be in writing, but looks like it is the reading that is causing the slow performance. Where are you reading the data from? Database? Then that's where the performance bottleneck is.