python fdb save huge data from database to file

Question

I have this script

SELECT = """
            select 
              coalesce (p.ID,'') as id, 
              coalesce (p.name,'') as name, 
            from TABLE as p  
         """
self.cur.execute(SELECT)
for row in self.cur.itermap():         
    id = '%(id)s' % row
    name = '%(name)s' % row

    xml +="  <item>\n"      
    xml +="    <id>"         + id + "</id>\n"
    xml +="    <name>"    + name + "</name>\n"    
    xml +="  </item>\n\n"
    
#save xml to file here
f = open...

  

and I need to save data from huge database to file. There are 10 000s (up to 40000) of items in my database and it takes very long time when script runs (1 hour and more) until finish.

How can I take data I need from database and save it to file "at once"? (as quick as possible? I don't need xml output because I can process data from output on my server later. I just need to do it as quickly as possible. Any idea?)

Many thanks!

P.S. I found out this interesting thing: When I use this code to "erase" xml variable every 2000 records and save it to another variable it works pretty fast! So there must be something "wrong" with filling in xml variable according to my former code.

result = float(id)/2000
if result == int(result):
  xml_whole += xml
  xml = ""

Answer 1

wow, after testing with code

result = float(id)/2000
if result == int(result):
  xml_whole += xml
  xml = ""

is my script up to 50x faster! i would like to know why is python so slow with xml +=... ?

Answer 2

You're doing a lot of unnecessary work (and however, if you erase the xml variable, you're not writing the same data as before...)

Why don't you just write the XML as it goes? You could also avoid the two COALESCEs, and write that check in Python (if ID is null then make id '', etc.).

SELECT = """
            select 
              coalesce (p.ID,'') as id, 
              coalesce (p.name,'') as name, 
            from TABLE as p  
         """
self.cur.execute(SELECT)

# Open XML file
f = open("file.xml", ...)
f.write("<?xml version... (what encoding?)

for row in self.cur.itermap():
    f.write("<item>\n    <id>%(id)s</id>\n    <name>%(name)s</name>\n</item>\n"

# Other f.writes() if necessary
f.close()