Requesting filesize and MIME type for a list of URLs in Android

Question

In an Android application, I have a list of image URLs like this:

List<String> urls = new ArrayList<String>(100);
urls.add("http://www.example.org/1.jpg");
urls.add("http://www.example.org/2.png");
urls.add("http://www.example.org/3.jpg");
urls.add("http://www.example.org/4.jpg");
urls.add("http://www.example.org/5.png");
urls.add("http://www.example.org/6.png");
urls.add("http://www.example.org/7.png");
urls.add("http://www.example.org/8.jpg");
urls.add("http://www.example.org/9.jpg");
urls.add("http://www.example.org/10.gif");
    ...
urls.add("http://www.example.org/100.jpg");

Now I have to get the filesize and MIME type for all of these URLs, and this should be done as fast as possible, of course.

What I did is the following:

for (String url : urls) {
    int fileSize;
    try {
        URLConnection urlConnection;
        urlConnection = new URL(url).openConnection();
        urlConnection.connect();
        final String mimeType = urlConnection.getContentType();
        final int fileSize = urlConnection.getContentLength();
        // do something with those two pieces of information
    }
    catch (MalformedURLException e) {
        continue;
    }
    catch (IOException e) {
        // some special handling
    }
}

But this is terribly slow. This is because it is using a single thread and requesting the URLs one by one, while a web browser would always access multiple files at a time, isn't it?

So how can I make it faster?

For HttpClient, I've read that you should re-use instances and there are some ways to use them in a multi-threaded environment.

But how would I do this with URLConnection or any other class that gives you filesize and MIME type?

Edit:

The images are not all on the same host, but spread across only a few servers, say 100 images spread across 5 host names.

Can you use a few threads or run several AsynTasks at once that do the job? Is there anything you have to pay attention to, such as recycling URLConnection objects or so?

I'm not quite sure how to use multiple threads to share task list (100 image files) and merge the results (MIME types and file sizes) afterwards. Can you help?

Answer 1

Split your work up into smaller peaces and let a worker Thread handle it:

The worker Thread:

public class Downloader extends Thread {

    private final String mUrl;
    private String mMimeType;
    private int mFileSize;

    public Downloader(String url) {
        mUrl = url;
    }

    @Override
    public void run() {
        try {
            URLConnection urlConnection;
            urlConnection = new URL(mUrl).openConnection();
            urlConnection.connect();
            mMimeType = urlConnection.getContentType();
            mFileSize = urlConnection.getContentLength();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public String getMimeType() {
        return mMimeType;
    }

    public int getFileSize() {
        return mFileSize;
    }
}

Instantiate, run and wait for the worker:

ArrayList<String> urls = new ArrayList<String>(10);
// ...
ArrayList<Thread> threads = new ArrayList<Thread>(10);
for (String url : urls) {
    Thread t = new Downloader(url);
    threads.add(t);
    t.start();
}
for (Thread t : threads) {
    try {
        // do not wait for other threads  in main UI thread!
        t.join();
        //((Downloader) t).getMimeType();
        //((Downloader) t).getFileSize();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}

Make sure to note wait for the worker Thread in your UI Thread.

The answer above should only be used for a small set of URLs. A ThreadPool may not be necessary because the Threads will wait for IO most of the time.

---

Here is your requested answer with a ThreadPool.

It's using the same Downloader class as the above example with only one change: Spawning Threads is done by the ThreadPool and the single tasks don't need to be a real Thread anymore. So let the Downloader implement a Runnable instead of extending a Thread:

public class Downloader implements Runnable {

Hopefully it's what you are looking for.

public class ThreadedDownloader {

    private static final int KEEP_ALIVE_TIME = 1;
    private static final TimeUnit KEEP_ALIVE_TIME_UNIT = TimeUnit.SECONDS;
    private static int NUMBER_OF_CORES = Runtime.getRuntime().availableProcessors();
    private LinkedBlockingQueue<Runnable> mDecodeWorkQueue = new LinkedBlockingQueue<Runnable>();
    private ThreadPoolExecutor mDecodeThreadPool = new ThreadPoolExecutor(NUMBER_OF_CORES,
            NUMBER_OF_CORES, KEEP_ALIVE_TIME, KEEP_ALIVE_TIME_UNIT, mDecodeWorkQueue) {
        @Override
        protected void afterExecute(Runnable r, Throwable t) {
            super.afterExecute(r, t);
            Downloader d = (Downloader) r;
            // do something with finished Downloader d
            // like saving it's result to some sort of list
            // d.getMimeType();
            // d.getFileSize();

            if (mDecodeWorkQueue.isEmpty()) {
                onAllDownloadsFinised();
            }
        }
    };

    /** Download a list of urls and check it's mime time and file size. */
    public void download(List<String> urls) {
        for (String url : urls) {
            mDecodeThreadPool.execute(new Downloader(url));
        }
    }

    /** Calles when all downloads have finished. */
    private void onAllDownloadsFinised() {
        // do whatever you want here
        // update UI or something
    }
}

Answer 2

I don't have example Code, but the HTTP verb HEAD is what you're looking for. It retrieves the headers (including mime and content-length) without transferring the content body.

This answer goes into more detail about what HEAD does.

Answer 3

I think you have most, if not all of the pieces to your solution in the answers already submitted.

Here's what I'd do:

1) make the head requests using Executors.newCachedThreadPool();

2) store the responses in a Map

3) create a payload Map

4) load the real images using AsyncTask instances

5) when each bitmap load completes, store the results in the payload map

Just my thoughts.