As I look back on this matter, I want to add some more info.
First off, the use case that was required by the application was very retarded and cumbersome (but hey, clients, what can you do...). So like Joe stated above, I wouldn't download data on Asyncs in a million years now. One should use some sort of Service for downloading the data required, if possible.
Secondly, I ended up using RoboSpice library (it also provides caching) instead of Asyncs. It's still not as good as running on a Service, but it's much more well optimised than the barebone version. Might wanna check that out.