Load testing a notification hub

Question

I have a requirement to deliver push notifications to an app that runs on iOS and Android, with approximately 2 million installations in total. I've built a PoC using Azure Notification Hubs. This works fine tested against a handful of phones / tablets I could borrow. I've also tried the same with Amazon's SNS and that worked well too.

I have no reason to believe that hubs wouldn't scale as I need it to but I wondered if there was any provision for load testing. I can't borrow 2m phones but maybe I could configure a hub to call a service I host, thereby simulating a push to either the GCM or APNS gateways? This would help build confidence in my end-to-end performance / volume testing.

Answer 1

I believe this is not supported. If there is a load testing capability, it's internal to Azure and not offered for public use.

However, Microsoft does provide an SLA for the Basic and Standard tiers of Notification Hubs. They claim they use the Notification Hubs service to deliver things like the Breaking News alerts for the Bing News apps. The SLA guarantees 99.9% successful message delivery within five minutes (over a month).

The Service Bus SLA (which covers Notification Hubs) is here: http://www.microsoft.com/en-us/download/details.aspx?id=4767

I could not find SLAs for GCM or APNS.

Notification Hubs do provide a fairly rich reporting API, that you can query with OData filters to determine how many notifications are sent over given periods of time.

But I expect that the variable load conditions that affect the service as a whole will mean that no specific promise is made about the specific timeliness of any delivery (within the five-minute guaranteed delivery time). In other words, all of your 2 million notifications might be delivered within fifteen seconds, or it might take 4 minutes to send the first message, with all of them delivered at 4.9 minutes, depending on who else is using the service and how heavily they are using it.