HttpUrlConnection multipart file upload with progressBar

Question

I want to check progress of uploading file by HttpUrlConnection. How I can do this? I've tried to calculate bytes when writing data in OutputStream but it's wrong, cause real uploading happens only when I call conn.getInputStream(), so I need somehow to check inputStream. Here is my code:

public static void uploadMovie(final HashMap<String, String> dataSource, final OnLoadFinishedListener finishedListener, final ProgressListener progressListener) {
  if (finishedListener != null) {
    new Thread(new Runnable() {
       public void run() {
         try {

              String boundary = getMD5(dataSource.size()+String.valueOf(System.currentTimeMillis()));
              MultipartEntityBuilder multipartEntity = MultipartEntityBuilder.create();
              multipartEntity.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);    
              multipartEntity.setCharset(Charset.forName("UTF-8"));

              for (String key : dataSource.keySet()) {
                 if (key.equals(MoviesFragmentAdd.USERFILE)) {
                    FileBody  userFile = new FileBody(new File(dataSource.get(key)));
                    multipartEntity.addPart(key, userFile);
                    continue;
                 }
                 multipartEntity.addPart(key, new StringBody(dataSource.get(key),ContentType.APPLICATION_JSON));
              }

              HttpEntity entity = multipartEntity.build();
              HttpURLConnection conn = (HttpsURLConnection) new URL(URL_API + "/video/addForm/").openConnection();
              conn.setUseCaches(false);
              conn.setDoOutput(true);
              conn.setDoInput(true);
              conn.setRequestMethod("POST");
              conn.setRequestProperty("Accept-Charset", "UTF-8");
              conn.setRequestProperty("Connection", "Keep-Alive");
              conn.setRequestProperty("Cache-Control", "no-cache");
              conn.setRequestProperty("Content-Type", "multipart/form-data;boundary=" + boundary);
              conn.setRequestProperty("Content-length", entity.getContentLength() + "");
              conn.setRequestProperty(entity.getContentType().getName(),entity.getContentType().getValue());

              OutputStream os = conn.getOutputStream();
              entity.writeTo(os);
              os.close();

              //Real upload starting here -->>

              BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));

              //<<--

              JsonObject request = (JsonObject) gparser.parse(in.readLine());
              if (!request.get("error").getAsBoolean()) {
              //do something
              }
              conn.disconnect(); 

           } catch (Exception e) {
            e.printStackTrace();
           }
         }
    }).start();

  }
}

Answer 1

Because you have to deal with upload, I'd suppose most time is taken when doing entity.writeTo(os);. Maybe the first contact to the server takes some time as well (DNS resolution, SSL-handshake, ...). The markers you set for "the real upload" are not correct IMO.

Now it depends on your Multipart-library, whether you can intercept writeTo. If it is clever and resource-efficient, it's iterating over the parts and streams the content one-by-one to the output stream. If not, and the .build() operation is creating a big fat byte[], then you could take this array, stream it in chunks to the server and tell your user how many percent of the upload is already done.

From a resource perspective I'd prefer not really knowing what happens. But if feedback is that important and if the movies are only a few megabytes in size, you could stream the Multipart-Entity first to a ByteArrayOutputStream and then write little chunks of the created byte-array to the server while notifying your user about progress. The following code is not validated or tested (you can see it as pseudo-code):

ByteArrayOutputStream baos = new ByteArrayOutputStream();
entity.writeTo(baos);
baos.close();
byte[] payload = baos.toByteArray();
baos = null;

OutputStream os = conn.getOutputStream();

int totalSize = payload.length;
int bytesTransferred = 0;
int chunkSize = 2000;

while (bytesTransferred < totalSize) {
    int nextChunkSize = totalSize - bytesTransferred;
    if (nextChunkSize > chunkSize) {
        nextChunkSize = chunkSize;
    }
    os.write(payload, bytesTransferred, nextChunkSize); // TODO check outcome!
    bytesTransferred += nextChunkSize;

    // Here you can call the method which updates progress
    // be sure to wrap it so UI-updates are done on the main thread!
    updateProgressInfo(100 * bytesTransferred / totalSize);
}
os.close();

A more elegant way would be to write an intercepting OutputStream which registers progress and delegates the real write-operations to the underlaying "real" OutputStream.

Edit

@whizzzkey wrote:

I've re-checked it many times - entity.writeTo(os) DOESN'T do a real upload, it does conn.getResponseCode() or conn.getInputStream()

Now it's clear. HttpURLConnection is buffering your upload data, because it doesn't know the content-length. You've set the header 'Content-length', but oviously this is ignored by HUC. You have to call

conn.setFixedLengthStreamingMode(entity.getContentLength());

Then you should better remove the call to conn.setRequestProperty("Content-length", entity.getContentLength() + "");

In this case, HUC can write the headers and entity.writeTo(os) can really stream the data to the server. Otherwise the buffered data is sent when HUC knows how many bytes will be transferred. So in fact, getInputStream() tells HUC that you're finished, but before really reading the response, all the collected data has to be sent to the server.

I wouldn't recommend changing your code, but for those of you who don't know the exact size of the transferred data (in bytes, not characters!!), you can tell HUC that it should transfer the data in chunks without setting the exact content-length:

conn.setChunkedStreamingMode(-1); // use default chunk size

Answer 2

Right this code in your activity...

public class PublishPostToServer extends AsyncTask implements ProgressListenerForPost {

    public Context pContext;
    public long totalSize;
    private String response;

    public PublishPostToServer(Context context) {
        pContext = context;

    }

    protected void onPreExecute() {
        showProgressDialog();
    }

    @Override
    protected Boolean doInBackground(Void... params) {
        boolean success = true;
        try {
            response = NetworkAdaptor.getInstance()
                    .upLoadMultipartImageToServer(
                            "",
                            "",
                            "", this, this); // Add file path, Authkey, caption 

        } catch (Exception e) {
            success = false;
        }
        return success;
    }

    @Override
    protected void onPostExecute(Boolean result) {
        super.onPostExecute(result);
        //validateResponse(result, response);
    }

    @Override
    protected void onProgressUpdate(Integer... values) {

        try {
            if (mProgressDialog != null) {
                mProgressDialog.setProgress(values[0]);
            }
        } catch (Exception exception) {

        }
    }

    @Override
    public void transferred(long num) {
        publishProgress((int) ((num / (float) totalSize) * 100));
    }

}

private void showProgressDialog() {

    try {
        String dialogMsg = "Uploading Image...";
        mProgressDialog = new ProgressDialog(this);
        mProgressDialog.setMessage(dialogMsg);
        mProgressDialog.setIndeterminate(false);
        mProgressDialog.setMax(100);
        mProgressDialog.setProgressStyle(ProgressDialog.STYLE_HORIZONTAL);
        mProgressDialog.setCancelable(false);
        mProgressDialog.show();
    } catch (Exception exception) {

    }
}

Now, Make a NetworkAdapter Class

public String upLoadMultipartImageToServer(String sourceFileUri, String auth_key, String caption, ProgressListenerForPost listiner, PublishPostToServer asyncListiner) { String upLoadServerUri = "" + "upload_image";

    HttpPost httppost = new HttpPost(upLoadServerUri);

    File file = new File(sourceFileUri);

    if (file.exists()) {

        FileBody filebodyVideo = new FileBody(file);
        CustomMultiPartEntity multipartEntity = new CustomMultiPartEntity(
                HttpMultipartMode.BROWSER_COMPATIBLE, listiner);
        try {
            multipartEntity.addPart("auth_key", new StringBody(auth_key));
            multipartEntity.addPart("caption", new StringBody(caption));
            multipartEntity.addPart("image", filebodyVideo);
            asyncListiner.totalSize = multipartEntity.getContentLength();
            httppost.setEntity(multipartEntity);

        }

        catch (UnsupportedEncodingException e1) {
            // TODO Auto-generated catch block
            e1.printStackTrace();
        }

        DefaultHttpClient mHttpClient = new DefaultHttpClient();
        String response = "";
        try {
            response = mHttpClient.execute(httppost,
                    new MovieUploadResponseHandler());
        } catch (ClientProtocolException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        return response;
    } else {
        return null;
    }

} 

@SuppressWarnings("rawtypes")
private class MovieUploadResponseHandler implements ResponseHandler {

    @Override
    public Object handleResponse(HttpResponse response)
            throws ClientProtocolException, IOException {

        HttpEntity r_entity = response.getEntity();
        String responseString = EntityUtils.toString(r_entity);
        // DebugHelper.printData("UPLOAD", responseString);

        return responseString;
    }

}

public static boolean isValidResponse(String resultData) {
    try {

    } catch (Exception exception) {
        //DebugHelper.printException(exception);
    }
    return true;
}

public String upLoadVideoToServer(String currentFilePath, String string,
        PublishPostToServer publishPostToServer,
        PublishPostToServer publishPostToServer2) {
    // TODO Auto-generated method stub
    return null;
}