There are 2 problems here which are contributing to the slowness here. The first is the small page size. There are internal discussions happening right now about bumping this to 1000 (especially since now in v2.0 collections aren't returned in the request).
The second is that right now WsapiDataStore does its paging serially. The first page always has to be loaded serially, but after that the remaining n-1 should be able to be done in parallel.
There is a defect tracking the second item that will hopefully be addressed soon. Hopefully the first item will get some attention as well.