Scala and Java futures apparently having unexpected interactions

Question

We are using Elasticsearch 0.90.7 in our Scala Play Framework application, where the end of our "doSearch" method looks like:

def doSearch(...) = {
  ...
  val actionRequessBuilder: ActionRequestBuilder // constructed earlier in the method
  val executedFuture: ListenableActionFuture<Response> = actionRequestBuilder.execute
  return executedFuture.actionGet
}

where ListenableActionFuture extends java.util.concurrent.Future, and ListenableActionFuture#actionGet is basically the same as Future#get

This all works fine when we execute searches sequentially, however when we try to execute multiple searches in parallel:

val search1 = scala.concurrent.Future(doSearch(...))
val search2 = scala.concurrent.Future(doSearch(...))
return Await.result(search1, defaultDuration) -> Await.result(search2, defaultDuration))

we're sometimes (less than 1 or 2% of the time) getting unexpected timeouts on our scala futures, even when using an extremely long timeout during qa (5 seconds, where a search always executes in less than 200ms). This also occurs when using the scala global execution context as well as when using the Play default execution context.

Is there some sort of unexpected interaction going on here as a result of having a java future wrapped in a scala future? I would have thought that the actionGet call on the java future at the end of doSearch would have prevented the two futures from interfering with each other, but evidently that may not be the case.

Answer 1

I thought it was established somewhere that blocking is evil. Evil!

In this case, Await.result will block the current thread, because it's waiting for a result.

Await wraps the call in blocking, in an attempt to notify the thread pool that it might want to grow some threads to maintain its desired parallelism and avoid deadlock.

If the current thread is not a Scala BlockContext, then you get mere blockage.

Whatever your precise configuration, presumably you're holding onto a thread while blocked, and the thunk you're running for search wants to run something and can't because the pool is exhausted.

What's relevant is what pool produced the current Thread: whether the go-between Future is on a different pool doesn't matter if, at bottom, you need to use more threads from the current pool and it is exhausted.

Of course, that's just a guess.

It makes more sense to have a single future that gets the value from both searches, with a timeout.

But if you wind up with multiple Futures, it makes sense to use Future.sequence and wait on that.