Should I take live migrations into account when tracking measuring performance of Amazon EC2 instances?

Question

One of our applications is running on a number of EC2 instances, and we dynamically add/remove instances based on load. We want to try to ensure that performance variation across the images stays within a certain (configurable) bound.

As has been demonstrated by various studies, especially [1], simply starting the same type of instance is often not sufficient if the acceptable variation is small:

"We demonstrated that the CPU and disk I/O performance of small instances are relatively stable from the perspective of a long-running periods. However, the performance behavior of multiple “identical" small instances is very heterogeneous."

As a result, we are now currently starting a few more instances then required, running a benchmark and keeping only those whose performance is acceptable. There's an assumption here that, if the instance performs adequately during the benchmark, its performance will be relatively stable.

This appears to be mainly due to the fact that, once an instance is started, it remains on the same underlying host until stopped or terminated (see e.g. [2]). However, there have been recent reports that EC2 has changed parts of the underlying Xen infrastructure and can now support live migration (internally, i.e. not exposed to the user) [3].

"You can infer with relative confidence that that means they’ve fully divorced the instance from the hardware and can therefore now live-migrate instances."

If Amazon does start to live-migrate instances across hosts, then of course benchmarking-on-startup isn't sufficient. So the question is: should we assume that Amazon could or will be live-migrating instances "under the covers" in the near future?

Thanks

Andrew

[1] "EC2 Performance Analysis for Resource Provisioning of Service-Oriented Applications", http://www.globule.org/publi/EPARPSOA_nfpsla2009.pdf

[2] Response of "JasonM@AWS", https://forums.aws.amazon.com/thread.jspa?messageID=198800𰢐

[3] http://default.io/2010/09/to-do-learn-ec2-and-ubuntu-at-the-same-time/

Answer 1

Just to follow up on this one...it appears the answer is "no". At least, not currently. Not confirmed by anyone from Amazon, but I guess they'd be unlikely to do that:

https://forums.aws.amazon.com/message.jspa?messageID=218102

Answer 2

You're asking the wrong place here. Only amazon knows if they actually intend to do live-migration, if they intend to inform people, and if there's any way to find out.

If you want to be safe, you could periodically re-benchmark - but I think you'd be better served by looking at metrics from your own processing to do load balancing; things like average time for request processed, etc. By looking at these on a continual basis you can more easily deal with occasional performance variations, which can happen on virtualized systems for other reasons than live migrations.