Sending multiple record with one call using hiredis

Question

I hope this list is right for asking questions about redis client "hiredis" . I want to achieve the same thing which I am doing below with redis client . As can be seen redis send 3 different record with one rpush call .

redis 127.0.0.1:6379> rpush test kemal erdem husyin

(integer) 3
redis 127.0.0.1:6379> lrange test 0 -1
1) "kemal"
2) "erdem"
3) "husyin"

In my project I use hiredis an example :

reply =  (redisReply*)(redisCommand(c, "RPUSH %s %s" , channelName,  message));

But Now I have a big log file which every line is being hold in a buff like char[][]; I need to send each line as different records but also need calling rpush only one time for performance .Would you have a advice for me ?

Answer 1

It would be a bad idea to send a unique command to push more than a few thousands of items. It would saturate the communication buffers, and a large command will block all other concurrent commands due to the single-threaded nature of Redis.

I suggest to build your push commands by batch of small packets of n items (n between 10 and 100), and to group your push commands in a pipeline of m commands (m between 10 and 100).

The algorithm would be something like this:

While there are still lines to read:
   New Redis pipeline, i=0
   While there are still lines to read and i<m:
      Read at most n lines
      Build push command for the read lines
      Pipeline push command
      ++i
   Flush Redis pipeline, check return status if needed

It will only generate N / (n*m) roundtrips (N being the number of lines in the input file).

To build commands with arbitrary numbers of parameters, you can use the redisAppendCommandArgv function.