Google App Engine instances have big spike despite normal traffic

Question

For the past few months my app engine app has run with 2 instances no problem. Occasionally I have upped this to 4 or 5 to account for heavy usage.

Then all of a sudden the number of instances shot up yesterday, reaching 250 before I maxed out my $25/day maximum budget and for the next 6 hours seved 503. Then it magically started serving again.

It is now running fine again with 2 instances. Unfortunately the outage came while I was fast asleep in NZ and I only noticed the unusual behavior on the instance chart this morning.

There have been NO traffic spikes in the last 24 hours. And there did not appear to be anything unusual in the logs that would have caused a spike to 250 instances.

At the time the instances started rising (-20hr on the chart) I did deploy an "alpha" version of my app, which FAILED to complete. I rolled it back, then redeployed and it worked fine the 2nd time. Could something to do with the failed deployment have caused this massive spike?

Any other ideas as to what I can do to work out what went wrong?

I have been using GAE for over 2 years and never had anything like this occur. It is quite worrying!

Answer 1

Im adding this as an answer as many people don't read the comments on the question.

The same thing happened to our app, it was an issue on App Engine's side apparently:

https://groups.google.com/forum/#!msg/google-appengine-downtime-notify/VmH7s-Eeyso/mgoxt1dMdFg