Solution found:
The Asynchronous Sessions cleanup phase starting NOW ...
was caused by some calls to external feeds, these were intermittently slow. This phase must have been waiting for these calls, which were timing out, but PhantomJS sat there waiting for a very, very long time.
Removing these calls when testing (with a wee bit of JavaScript, if host is localhost then use a dummy object) solved the problem.