can not select ten million rows from a database using python

Question

I am currently attempting to load ten million rows of data using python library pyodbc. Consider the following example where the customer table has over ten million rows, please note jeetl :

results = jeetl.dw_command_cursor("select * from Customer")

The above works fine if i limit it to just one million rows but falls flat on its face with ten, is there a workaround for this? Here is the error message i have been seeing:

pyodbc.Error: ('08S01', '[08S01] Communication error while getting a tuple;\nCommunication    error while getting a tuple (26) (SQLExecDirectW)')

thanks in advance

Answer 1

Maybe your query result is too large, try checking the max_allowed_packet_size. For example in mysql: https://dev.mysql.com/doc/refman/5.5/en/packet-too-large.html

Maybe what you need to do is paging the results: http://www.petefreitag.com/item/451.cfm