Redis insertion to hash is VERY(!) slow?

Question

I have a jagged array (1M x 100) of random numbers :

0   --->[ 100 random numbers]
1   --->[ 100 random numbers]
2   --->[ 100 random numbers]
..  --->[ 100 random numbers]
..  --->[ 100 random numbers]
1M  --->[ 100 random numbers]

Those 100 random numbers are Images ID's which I need to map them to userId (which is 0..1M)

I want to store them in Redis. ( via Hash since I know that Hashes are stored very efficiently in Redis — it's also how instagram used it in order to solve similar problem)

---

The Jagged array :

int b = 1000000;
int[ ][] Friends = new int[b][];

(All ok.)

The initialization of random numbers :

  for (int i = 0; i < b; i++)
    {
     Friends[i] = ParallelEnumerable.Range(0, 100).Select(n => appRandom.Value.Next(1, b )).ToArray();
    }

(All ok.)

And here is How I use Redis Client to enter values : (I use 1000 buckets optimization method)

  using (IRedisClient redisClient = new RedisClient(host))
     {
       for (int i = 0; i < Friends.Length; i++)
         {
            for (int j = 0; j < 100; j++)
             redisClient.SetEntryInHash("urn:buck>" + (i / 1000), Friends[i][j].ToString(),i.ToString());
         }
     }

Problem :

The Insertion phase( I don't care about the initialization time)— It is taking over an hour now and still not at 25% !

related info :

I have a powerful computer : (win 8 ) , Also added the <gcAllowVeryLargeObjects enabled="true" />

Question :

• What am I doing wrong and how can I make the insertion faster ?

(related info : those 100 numbers are Images Id's which I need to map them to userId)

Answer 1

Please read http://redis.io/topics/benchmarks

Your peformance bottleneck is probably related to the number of roundtrips between your application and the redis server. You need to use pipelining, or use concurrent connections, or both, in order to maximize the throughput.

See examples of pipelining usage at: https://github.com/ServiceStack/ServiceStack.Redis/blob/master/tests/ServiceStack.Redis.Tests/RedisPipelineTests.cs

Last point: Windows is certainly not the best platform to get the best performance from Redis, and C# not the best language to achieve very high throughput.