Execute over 1000 insert statements in 1 string

Question

I thought this one would be simple, but for whatever reason the data isn't in the table when the script is done. I am generating a string of over 1000 queries, currently 1131, and they are separated by ";".

This string is created by a simple for loop:

query = ""

for: #loop condition
    query += "INSERT INTO Results (col1, col2, col3, col4, pass) VALUES ('%s', %s, %s, %s, 0);" % (val1, val2, val3, val4)
logger.debug("SQL Query = %s" % query)

try:
    cursor.execute(query, multi=True)
except MySQLdb.Error, e:
    try:
        logger.error("MySQL Error [%d]: %s" % (e.args[0], e.args[1]))
        dbinit.cnx.rollback()
        cursor.close()
        dbinit.cnx.close()
    except IndexError:
        logger.error("MySQL Error: %s" % str(e))
        dbinit.cnx.rollback()
        cursor.close()
        dbinit.cnx.close()

dbinit.cnx.commit()
cursor.close()
dbinit.cnx.close()

I'm not getting any errors, and when I print the string, it prints just fine. I can even copy/paste whats printed, and run the sql commands all at once and it runs just fine, so I don't have any bad queries. Any help is appreciated, because I am stumped. I know there are alternate ways to go about this, but if someone could also explain why this one isn't working, so I can learn, that would be appreciated too!

Update: Mike's answer is right for the most part, but I just wanted to update this post with what my code ended up looking like in the end.

queryData=[]
for: #condition
    queryData.append((val1, val2, val3, val4, 0))

while len(queryData) != 0:
    cursor.executemany("INSERT INTO Results (col1, col2, col3, col4, pass) VALUES (%s, %s, %s, %s, %s)", queryData[:999])
    del queryData[:999]
dbinit.cnx.commit()

Answer 1

It looks like you'r tackling this the wrong way. cursor.execute() takes one query, prepares it and executes it with the data you provide. In this line

cursor.execute(query, multi=True)

You've omitted the second params argument, so there's no data to execute with.

You seem to be assembling 1000+ queries as a single string. You could perhaps do this with cursor.query(), but that (if it works) will cause the server to parse and prepare essentially the same query 1000+ times, which is less than efficient. In any case, many MySQL classes specifically prohibit assembling multiple queries this way.

What you need is cursor.executemany(query, data) which constructs a single query, with multiple value sets, one for each row of yourdata.

I'm not a Python developer, and I cant see where you're getting your data from, but this is the example from the MySQL reference, reworked:

data = [
  ('col1-1', 1,2,3),
  ('col1-2', 4,5,6),
  ('col1-3', 7,8,9),
]
stmt = "INSERT INTO Results (col1, col2, col3, col4, pass) VALUES ('%s', %s, %s, %s, 0)"
cursor.executemany(stmt, data)

You can add your own exception handling. Hopefully, that will get you started.

Note: there's no limit to the number of value sets you can insert this way, but there is a limit to the length of the string you can create, which by default is 1Mb. If your requirements exceed this you could batch the inserts to, say, 500 at a time and run several queries (3 queries is still better than 1500!) or look at the cursor.MySQLCursorPrepared class which parses the query once and executes for each set of data you send.