Redis php processing array part by part (may run out of memory)

Question

For example i have list in redis with some data, this list can be really large so if i use getList method i can run out of memory.

//getList
$Data = Rediska()->getList('example_list_of_data');
foreach ($Data as $value) {
    if (in_array($someValue, $value)) {
        //delete element from list
        Rediska()->popFromList('example_list_of_data', $value);
    }
}

I dont want to get full list from Redis, i want to get data part by part(for exmaple of 50 pieces). Like so:

   get 50 first items
   ->try to find element
   ->if so delete it and stop iteration
   ->if did not find iterate next 50 etc...

how can i achive this? i can get list size with LLEN

Answer 1

i have found solution, get list part by part, tested it with microtime and memory usage. It takes a little bit more time, but much less memory on list with 1000000 records.

public static function processRedisList($Start, $End, $ListName
    $ElementToFind, $Found = false, $Part
){
   while($Found != true) {
     //
     $Data = NoSQL_Singleton::Rediska()->getList($ListName, $Start, $End);
     if (!empty($Data)) {
        foreach ($Data as $Key => $Value) {
           //process List here
           //do what you want
           $Found = true;
           return $Found;
        }
      $Start = $Start + $Part;
      $End = $End + $Part;
    }
   }
   return false;
}

Answer 2

You could try to use the toArray method of the Rediska_Key_List. It gets elements from the list. The first parameter of this method is the start index, the second the end index. From the documentation:

array toArray ([integer $start = 0], [integer $end = -1], [ $responseIterator = false], boolean $responseIterator[optional])

integer $start: Start index 
integer $end: End index 
boolean $responseIterator[optional]: If true - command return iterator which read from socket buffer. Important: new connection will be created 
$responseIterator