Python mysqldb fetchmany not working properly

Question

I have some code like this

if __name__ == '__main__':
    def result_generator(cursor, batch_size=10):
        while True:
            results = cursor.fetchmany(batch_size)
            if not results:
                break
            for res in results:
                yield res

    db = MySQLdb.connect(host="localhost", user="root", passwd="root", db="domains")
    # you must create a Cursor object. It will let
    #  you execute all the queries you need
    cursor = db.cursor()
    cursor.execute("SELECT domain FROM com ORDER BY id ASC")
    for result in result_generator(cursor):
        url = "http://www.{0}".format(result[0])
        print url
        w = Wappalyzer(url)
        out = w.analyze()
        cursor.execute("""UPDATE com SET frameworks=%s, is_checked=1 WHERE domain=%s""",
                       (db.escape_string(out.get('frameworks', "")), result[0]))
    # disconnect from server
    db.commit()
    db.close()

My result_generator fetches domain names from the database in batches. Batch size is 10.

My program working fine IF I remove this line

cursor.execute("""UPDATE com SET frameworks=%s, is_checked=1 WHERE domain=%s""",
                           (db.escape_string(out.get('frameworks', "")), result[0]))

Otherwise it runs only the first batch.

Can someone tell me whats wrong with my code?

Answer 1

Off the top of my head, could it be that the result in cursor is overwritten at the next execute call?

Your SELECT call fills the results. The first call to the results generator works fine because there are still results from the SELECT. You then use the same cursor to execute an UPDATE, which "resets" the cursor results. Hence, the next call to the generator kills the loop. If you remove the UPDATE, the cursor results aren't reset, allowing the loop to continue.

This is just a guess. I haven't tried it out.