Azure Storage Queue very slow from a worker role in the cloud, but not from my machine

Question

I'm doing a very simple test with queues pointing to the real Azure Storage and, I don't know why, executing the test from my computer is quite faster than deploy the worker role into azure and execute it there. I'm not using Dev Storage when I test locally, my .cscfg is has the connection string to the real storage.

The storage account and the roles are in the same affinity group.

The test is a web role and a worker role. The page tells to the worker what test to do, the the worker do it and returns the time consumed. This specific test meassures how long takes get 1000 messages from an Azure Queue using batches of 32 messages. First, I test running debug with VS, after I deploy the app to Azure and run it from there.

The results are:

• From my computer: 34805.6495 ms.
• From Azure role: 7956828.2851 ms.

That could mean that is faster to access queues from outside Azure than inside, and that doesn't make sense.

I'm testing like this:

    private TestResult InQueueScopeDo(String test, Guid id, Int64 itemCount)
    {
        CloudStorageAccount account = CloudStorageAccount.Parse(_connectionString);
        CloudQueueClient client = account.CreateCloudQueueClient();
        CloudQueue queue = client.GetQueueReference(Guid.NewGuid().ToString());

        try
        {
            queue.Create();
            PreTestExecute(itemCount, queue);

            List<Int64> times = new List<Int64>();
            Stopwatch sw = new Stopwatch();
            for (Int64 i = 0; i < itemCount; i++)
            {
                sw.Start();
                Boolean valid = ItemTest(i, itemCount, queue);
                sw.Stop();
                if (valid)
                    times.Add(sw.ElapsedTicks);
                sw.Reset();
            }

            return new TestResult(id, test + " with " + itemCount.ToString() + " elements", TimeSpan.FromTicks(times.Min()).TotalMilliseconds,
                                                 TimeSpan.FromTicks(times.Max()).TotalMilliseconds,
                                                 TimeSpan.FromTicks((Int64)Math.Round(times.Average())).TotalMilliseconds);
        }
        finally
        {
            queue.Delete();
        }

        return null;
    }

The PreTestExecute puts the 1000 items on the queue with 2048 bytes each.

And this is what happens in the ItemTest method for this test:

    Boolean done = false;
    public override bool ItemTest(long itemCurrent, long itemCount, CloudQueue queue)
    {
        if (done)
            return false;

        CloudQueueMessage[] messages = null;

        while ((messages = queue.GetMessages((Int32)itemCount).ToArray()).Any())
        {
            foreach (var m in messages)
                queue.DeleteMessage(m);
        }

        done = true;

        return true;
    }

I don't what I'm doing wrong, same code, same connection string and I got these resuts.

Any idea?

UPDATE:

The problem seems to be in the way I calculate it.

I have replaced the times.Add(sw.ElapsedTicks); for times.Add(sw.ElapsedMilliseconds); and this block:

return new TestResult(id, test + " with " + itemCount.ToString() + " elements",
TimeSpan.FromTicks(times.Min()).TotalMilliseconds,                             
TimeSpan.FromTicks(times.Max()).TotalMilliseconds,     
TimeSpan.FromTicks((Int64)Math.Round(times.Average())).TotalMilliseconds);

for this one:

return new TestResult(id, test + " with " + itemCount.ToString() + " elements", 
times.Min(),times.Max(),times.Average());

And now the results are similar, so apparently there is a difference in how the precision is handled or something. I will research this later on.

Answer 1

The problem apparently was a issue with different nature of the StopWatch and TimeSpan ticks, as discussed here.

Stopwatch.ElapsedTicks Property

Stopwatch ticks are different from DateTime.Ticks. Each tick in the DateTime.Ticks value represents one 100-nanosecond interval. Each tick in the ElapsedTicks value represents the time interval equal to 1 second divided by the Frequency.

Answer 2

How is your CPU utilization? Is this possible that your code is spiking the CPU and your workstation is much faster than your Azure node?