How to handle big arrays?

Question

I am developing an application in PHP for which I need to implement a big file handler. Reading and writing the file is not a problem, but checking the content of the file is a problem.

I built a recursive function which checks whether or not a variable is already used in the same document.

private function val_id($id){
    if(!isset($this->id)){
            $this->id = array();
        }
    if(in_array($id, $this->id)){
        return $this->val_id($id+1);
    }else{
        $this->id[] = $id;
        return $id;
    }
}

When in_array($id,$this->id) returns FALSE, the $id will be added to $this->id (array which contains all used ids) and returns a valid id.

When this returns TRUE, it returns the same function with parameter $id+1

Since we are talking about over 300000 records a time, PHP won't not to be able to store such big arrays. It seems to quit writing lines in the documents I generate when this array gets too big. But I don't receive any error messages like that.

Since the generated documents are SQL files with multiple rows INSERT another solution could be to check if the id already exists in the database. Can MySQL catch these exceptions and try these entries again with adding 1 to id? How?

How do you think I need to solve this problem?

Kind regards,

Wouter

Answer 1

Use INSERT IGNORE to disable duplicate key check in mysql and remove your key check in php. Your statement could look like this.

INSERT IGNORE INTO tbl_name SET key1 = 1, col1 = 'value1'

If you want to add 1 to the id always you could use ON DUPLICATE KEY to increment your key by one:

INSERT INTO table (a,b,c) VALUES (1,2,3)
    ON DUPLICATE KEY UPDATE c=c+1;

Answer 2

1. make error messages to appear.
2. increase memory_limit
3. instead of values store the parameter in the key - so you'll be able to use isset($array[$this->id]) instead of in_array()

Answer 3

Why should 30.000 records be a problem? Each record in a standard PHP array takes 144 bytes, for 30.000 that would mean 4218,75 kByte. No big deal.

Otherwise, Your Common Sense's idea with the array-key is worth a thought, because it's faster.