Node Redis: How to filter a sorted set of keys and retrieve each keys hashed values in one call

Question

I am working with a redis database in node.js using node_redis. Here is a quick example of a structure similar to what I am using.

hmset('user:1234', 
    'user_id', 1234, 
    'user_name', billy, 
    'user_age', 16);
//add user to group 1 store their id with their age as their score
zadd(['group:1:users_by_age', 16, user:1234]);

hmset('user:1235', 
    'user_id', 1235, 
    'user_name', jake, 
    'user_age', 21);
//add user to group 1 store their id with their age as their score
zadd(['group:1:users_by_age', 21, user:1235]);

Now lets say I wanted to get all user data for users over the age of 18 in group:1

I know I can get the user keys by calling

postClient.zrangebyscore( 
    [ 'group:1:users_by_age', '18', '+inf'],
    function( err, results ){
        console.log(results);
    }
);

Where I get lost is how do I fetch all the user objects at once? To take it one step further, is it possible to both zrangebyscore and get all the userobjects in one call?

Answer 1

To take it one step further, is it possible to both zrangebyscore and get all the userobjects in one call?

I don't believe you can. The SORT command has GET functionality built in which allows you to do such a thing in one call, but there's no way to pipe the results of a ZRANGEBYSCORE into SORT (barring storing it in a temporary key and then using SORT on that key).

There's also no natural way to retrieve multiple hashes in one call.

With your current implementation, considering these limitations, you might get all the users with a multi, like:

postClient.zrangebyscore([...], function (err, results) {
  var multi = postClient.multi();
  for (var i=0; i<results.length; i++){
    multi.hgetall(results[i]);
  }
  multi.exec(function(err, users){
    console.log(users);
  });
});

---

You could do this with a luascript though, retrieving the list of keys and iterating over it, calling hmget or hgetall on each key.

I do something like this in the following example, using hmget and specific keys.

Obvious disclaimer: I am not a lua programmer. Brief explanation: the script takes a start and end range for user age, then any number of hash keys which it uses for hmget on each user key, and appending it all to an array which will be wrapped up as user objects back in the javascript.

var script = '\
local keys = redis.call("ZRANGEBYSCORE", KEYS[1], ARGV[1], ARGV[2]); \
if not keys then return {} end; \
local users, attrs = {}, {} \
for i=3,#ARGV do \
  table.insert(attrs, ARGV[i]) \
end \
for i,key in pairs(keys) do \
  local vals = redis.call("HMGET", key, unpack(attrs)); \
  if vals then \
    for j,val in pairs(vals) do \
      table.insert(users, val) \
    end \
  end \
end \
return users \
';

// The rest of this you'd probably want to wrap up in an async function, 
// e.g `getUsersByRange`

// specify the user attributes you want
var attrs = ["id", "name", "age"];

// specify the range
var range = [18, "+INF"];

// get the attributes length, used to build the user objects
var attrlen = attrs.length;

// wrap up the params
var params = [script, 1, "users_by_age"].concat(range).concat(attrs);

// retrieve the user attributes in the form of [a1, a2, a3, a1, a2, a3, ... ],
// then collate them into an array of user objects like hgetall would return.
postClient.eval(params, function (err, res) {
  var users = [], i, j, k;
  for (i=0, j=0; i<res.length; i+=attrlen, j++) {
    users[j] = {};
    for (k=0; k<attrlen; k++) {
      users[j][attrs[k]] = res[i+k];
    }

    // this should be a list of users
    console.log(users);
  }
});

Note that it would be trivial to process the users back into objects inside the lua script, but when being converted back to redis data structures, lua tables become redis multi bulk replies (arrays), and the structure would be lost. Because of that it's necessary to convert the multi bulk reply into user objects back in javascript.