Scala: NoSuchElementException in for comprehension

Question

I have a future result of a web request that I need to count the size of. If the current response has items, I need to make another request to get the next set, etc. If the current response is empty, I'm done.

The code I have now:

def list(prefix: String, lastItem: Option[String] = None, last: Seq[BucketItem] = Nil):  Future[Iterable[BucketItem]] = {
  Logger.debug(s"list :: prefix=$prefix, lastItem=$lastItem, lastItems=${last.size}")
  for {
    current <- s3.get(name, None, Some(prefix), delimiter, lastItem, None) map listResponse // type is Seq[BucketItem]
    next <- list(prefix, Some(current.last.name), last ++ current) if !(current.isEmpty)
  } yield last ++ current
}

This seems to work fine until current doesn't have any more elements, and I get a NoSuchElementException trying to get current.last.name

I understand the condition if !(current.isEmpty) expands to a filter so it's not what I really want here. What I want is:

sequentially:
eval current
if current has more items, recursively call list to get the next set of items
yield the concatenated sequence of the right type (all the previous entries plus the last0

I'm using the for comprehension here to deal with collecting futures easily (at least this is the way I've done it in the past). Any guidance/things to read? I'm fairly new to scala, so please be gentle.

Answer 1

You don't use the variable next so there is no way you can get the result of s3.get calls after the first one. Also, your if !(current.isEmpty) will invoke withFilter on a Future value, which could produce a failed Future, which is not what you want here.

Here is a simpler solution:

def list(prefix: String, lastItem: Option[String] = None, accum: Seq[BucketItem] = Vector()):  Future[Iterable[BucketItem]] = {
  s3.get(name, None, Some(prefix), delimiter, lastItem, None) flatMap { current =>
    if (current.isEmpty)
      Future.successful(accum)
    else
      list(prefix, bucketItems.last.name, accum ++ current)
  }
}

A for-comprehension can only represent a map nested inside of flatMap calls (or foreach calls inside of each other) with interspersed calls to withFilter. All we need here is a single call to flatMap, so there is no use for for.

I have also replaced List with Vector, it performs better when appending elements to the end.

Answer 2

I think your filter is in the wrong place, this should work:

def list(prefix: String, lastItem: Option[String] = None, last: Seq[BucketItem] = Nil):  Future[Iterable[BucketItem]] = {
  Logger.debug(s"list :: prefix=$prefix, lastItem=$lastItem, lastItems=${last.size}")
  for {
    current <- s3.get(name, None, Some(prefix), delimiter, lastItem, None) map listResponse // type is Seq[BucketItem]
    if !current.isEmpty
    next <- list(prefix, Some(current.last.name), last ++ current)
  } yield last ++ current
}