DownloadManager downloading files over 2.1 GB

Question

I am working on an app and one of the features I am working on is to download some binary files. Some of them are really big (more than several mega-bytes). Downloads are completing fine as long as the file size is less than 2 GB.

I got stuck on a file that is 3.2GB in that I get progress updates (I am pooling the DownloadManager for progress updates), but when the download completes, the file is not present on the target file path. Interrogating the DownloadManager for that download id, I get STATUS_FAILED and reason ERROR_UNKNOWN - the favorite error details one will ever wish for!

What is weird is that this appears on most of the devices, but for some (like Samsung SG 4 Active OS 4.2.2 and LG Nexus 5 OS 4.4.2), it doesn't appear.

Doing some extra investigation, I found out that this seems to be a bug in Android DownloadManager implementation. It seems Android implementation stores the download count as an int, but when that count goes above Integer.MAX_VALUE the download ends as failed.

I am thinking to replace the DownloadManager usage with a foreground service, but I wouldn't give up yet ....

Did you guys face this and if so, how did you fix it? Is there any work-around to use DownloadManager in pre-4.2.2 so I can download more than 2.1 GB per file?

Answer 1

To download such a large files, you need to download those in chunks. Either you can use any library that support HTTP range options to allow to pull down a single file in multiple pieces , supporting resume etc.

Or you can split your large file on your server then have a text file with MD5 hash of each file, when you first start to download then get the MD5 file once finish then check that hashes matches the downloaded pieces. If they do not then delete that piece and add it to queue of items to download.

Once all pieces downloaded and MD5 works, you can put the pieces back together as single file.

If you are thinking to download the file in the SD card then FAT32 is the default file system. There is a 4 GB per file limit with this file system.

Answer 2

From looking at the Android source code, it appears that this issue was resolved in JB-MR2.

It seems that the only way to work around this on older platform versions would be to modify the server such that it uses chunked transfer encoding[1] for these large resources. In that case, that Download Manager will ignore and not attempt to parse the Content-Length header.

[1] http://en.wikipedia.org/wiki/Chunked_transfer_encoding

Answer 3

There is one clear outcome of this:

You can not fix the DownloadManager. If it's a bug in it, it will be so. Therefore, in short, No, you can not workaround this issue using the DownloadManager. You could however workaround it using a server side approach that has been put into words in the other answers.

So, I think your simplest solution would be to force the minimum sdk level to JB-MR2 because @ksasq mentioned that this issue has been resolved.

If that is not plausible nor in your case possible, you can find the best file download library out there and create an interface similar to DownloadManager's for this library. Of course, this interface should be implemented to use the default DownloadManager for versions which do not have this bug and use the custom library for those which had this bug (and for files who cause the issue if possible).

Unfortunately, a search on google showed yingyixu's android-download-manager last updated in 2012.

Another unfortunate note about this topic by CommonsWare simply verifies that there is no DownloadManager in google's support libraries. Worse is that the guy gave up the idea of implementing his own port becuase it was way too complicated. You can only hope that yingyixu's library or some other library you hopefully find is good enough.

Answer 4

You can pass this issue by splitting file into smaller zip files. Next step is to join them on target, I've found ->this<- that might help you. If you will not compress file (split only option) you should have good performance. Other issue is that you will need twice as much storage space. You can download smaller files, about 100MB, write it to joined buffer and remove form file-system, that will preserve space wasting.

Answer 5

You could also take the fixed version of DownloadManager, change the package to your package structure and use this version instead of the system version. Eventually you need to import some classes from the original package android.app. Then register your implementation as a service.