Odd behavior when fetching 100K rows from MySQL via PHP

Question

Looking for some ideas here... I have a MySQL table that has 100K rows of test data in it.

I am using a PHP script to fetch rows out of that table, and in this test case, the script is fetching all 100,000 rows (doing some profiling and optimization for large datasets).

I connect to the DB, and execute an unbuffered query:

$result = mysql_unbuffered_query("SELECT * FROM TestTable", $connection) or die('Errant query:  ' . $query);

Then I iterate over the results with:

if ($result) {
    while($tweet = mysql_fetch_assoc($result)) {
        $ctr++;

        if ($ctr > $kMAX_RECORDS) {
            $masterCount += $ctr;
            processResults($results);
            $results = array();
            $ctr = 1;
        }

        $results[] = array('tweet' => $tweet);
    }

    echo "<p/>FINISHED GATHERING RESULTS";
}

function processResults($resultSet) {
    echo "<br/>PROCESSED " . count($resultSet) . " RECORDS";
}

$kMAX_RECORDS = 40000 right now, so I would expect to see output like:

PROCESSED 40000 RECORDS
PROCESSED 40000 RECORDS
PROCESSED 20000 RECORDS

FINISHED GATHERING RESULTS

However, I am consistently seeing:

PROCESSED 39999 RECORDS
PROCESSED 40000 RECORDS

FINISHED GATHERING RESULTS

If I add the output of $ctr right after $ctr++, I get the full 100K records, so it seems to me to be some sort of timing issue or problem with fetching the data from the back-end with MYSQL_FETCH_ASSOC.

On a related note, the code in the while loop is there because prior to breaking up the $results array like this the while loop would just fall over at around 45000 records (same place every time). Is this due to a setting somewhere that I have missed?

Thanks for any input... just need some thoughts on where to look for the answer to this.

Cheers!

Answer 1

When you start going through your result $ctr has no value and doing first incrementation will evaluate it to 0. But when reaching $kMAX_RECORDS you reset it to 1 instead of 0. I don't know however why you see 1 row less in the first time of calling processResults(). I think it should be one more.

As to missing last 20000 rows notice that you are running processResults() only after $ctr exceeds $kMAX_RECORDS

Answer 2

You're building an array of results, and counting that new array's members. So yes, it is expected behavior that after fetching the first row, you'll get "1 result", then "2 results", etc...

If you want to get the total number of rows expected, you'll need to use mysql_num_rows()