Testing with 100's of devices

Question

We are building software intended to communicate with 100's or 1000's of PCs. Unfortunately we do not have the means to set that many devices up. We don't have that many physical devices, and we don't have enough infrastructure to support that many virtual devices either. We are looking to test with high volumes of PC's, combined with other factors such as network latency. Are there services or other ways we can achieve this level of testing?

Answer 1

There are cloud load testing services out there that might do what you want. A few I know of off hand are LoadStorm and Load Impact. Quite a few others will turn up with a search for something like "cloud load testing". This could be an easy option. There would be some cost, but it wouldn't be too high.

If you want to roll your own solution for free, a lot of infrastructure-as-a-service providers offer a free tier for new users. Amazon EC2 and Microsoft Azure both offer 750 hours of their smallest instance a month for free. While this is usually used for a whole contiguous month of server time (24 hours * 31 days), you could also use it to spin up 750 servers for an hour, once a month. Spread them across all the different regions/data centers available to maximize variance in networks and latency.

You could also consider writing a testing tool using a language with good concurrency support, or with a light footprint, so that you can fire up several hundred threads/processes at once, and then run your tests on relatively few servers. It wouldn't be quite the same as 1000s of different IP addresses all at once, but 4-5 servers each fielding a few hundred clients might be enough to satisfy your testing needs.