I have a jagged array (1M x 100
) of random numbers :
0 --->[ 100 random numbers]
1 --->[ 100 random numbers]
2 --->[ 100 random numbers]
.. --->[ 100 random numbers]
.. --->[ 100 random numbers]
1M --->[ 100 random numbers]
Those 100 random numbers
are Images ID's which I need to map them to userId (which is 0..1M
)
I want to store them in Redis. ( via Hash since I know that Hashes are stored very efficiently in Redis — it's also how instagram used it in order to solve similar problem)
The Jagged array :
int b = 1000000;
int[ ][] Friends = new int[b][];
(All ok.)
The initialization of random numbers :
for (int i = 0; i < b; i++)
{
Friends[i] = ParallelEnumerable.Range(0, 100).Select(n => appRandom.Value.Next(1, b )).ToArray();
}
(All ok.)
And here is How I use Redis Client to enter values : (I use 1000 buckets optimization method)
using (IRedisClient redisClient = new RedisClient(host))
{
for (int i = 0; i < Friends.Length; i++)
{
for (int j = 0; j < 100; j++)
redisClient.SetEntryInHash("urn:buck>" + (i / 1000), Friends[i][j].ToString(),i.ToString());
}
}
Problem :
The Insertion phase( I don't care about the initialization time)— It is taking over an hour now and still not at 25%
!
related info :
I have a powerful computer : (win 8 ) , Also added the <gcAllowVeryLargeObjects enabled="true" />
Question :
- What am I doing wrong and how can I make the insertion faster ?
(related info : those 100 numbers are Images Id's which I need to map them to userId)