Frage

I'm writing a webspider and allocating it on Heroku and after the main actor (the one that allocates the router) reaches 512Mb it crashes.

Basically every URL parsed is sent to the FetchRouter. The list of messages of the queue is to big to get in memory.

Is there any way to store this list in the Disk ?

fetcherRouter = context.actorOf(Props[URLFetcher].withRouter(RandomRouter(30)), name = "fetcherRouter")
War es hilfreich?

Lösung

Indeed.

There is FileBasedMailbox. Look in section 7.1, "Durable Mailboxes," of the Akka 2.0.x PDF (or corresponding Web page).

Lizenziert unter: CC-BY-SA mit Zuschreibung
Nicht verbunden mit StackOverflow
scroll top