Android to computer FTP resuming upload strange phenomenon

Question

I have a strange phenomenon when resuming a file transfer.

Look at the picture below you see the bad section.

This happens apparently random, maybe every 10:th time.
Im sending the picture from my Android phone to java server over ftp.

What is it that i forgot here.
I see the connection is killed due to java.net.SocketTimeoutException:
The transfer is resuming like this

Resume at : 287609 Sending 976 bytes more

The bytes are always correct when file is completely received. Even for the picture below.

Dunno where to start debug this since its working most of the times.

Any suggestions or ideas would be grate i think i totally missed something here.

The device Sender code (only send loop):

int count = 1;    
  //Sending N files, looping N times
  while(count <= max) {        
    String sPath = batchFiles.get(count-1);

    fis = new FileInputStream(new File(sPath));

    int fileSize =  bis.available();

    out.writeInt(fileSize); // size

    String nextReply = in.readUTF();
    // if the file exist,
    if(nextReply.equals(Consts.SERVER_give_me_next)){
        count++;                        
        continue;
    }
    long resumeLong = 0; // skip this many bytes 
    int val = 0;
    buffer = new byte[1024];

    if(nextReply.equals(Consts.SERVER_file_exist)){
        resumeLong = in.readLong();
    }

    //UPDATE FOR @Justin Breitfeller, Thanks
    long skiip = bis.skip(resumeLong);
if(resumeLong != -1){
   if(!(resumeLong == skiip)){
      Log.d(TAG, "ERROR skip is not the same as resumeLong ");
      skiip = bis.skip(resumeLong);
      if(!(resumeLong == skiip)){
        Log.d(TAG, "ERROR ABORTING skip is not the same as resumeLong);
        return;
      }
  }
    }

    while ((val = bis.read(buffer, 0, 1024)) > 0) {
        out.write(buffer, 0, val);
        fileSize -= val;
            if (fileSize < 1024) {
            val = (int) fileSize;
        }

    }

    reply = in.readUTF();
    if (reply.equals(Consts.SERVER_file_receieved_ok)) {
        // check if all files are sent
        if(count == max){
           break;
        }
    }
    count++;



   }

The receiver code (very truncated):

     //receiving N files, looping N times
    while(count < totalNrOfFiles){

        int ii = in.readInt(); // File size
        fileSize = (long)ii;

        String filePath = Consts.SERVER_DRIVE + Consts.PTPP_FILETRANSFER;
        filePath = filePath.concat(theBatch.getFileName(count));
        File path = new File(filePath);
        boolean resume = false;

        //if the file exist. Skip if done or resume if not
        if(path.exists()){
            if(path.length() == fileSize){ // Does the file has same size
                logger.info("File size same skipping file:" +                            theBatch.getFileName(count) );
                count++;
                out.writeUTF(Consts.SERVER_give_me_next);
                continue;   // file is OK don't upload it again
            }else { 
                // Resume the upload
                out.writeUTF(Consts.SERVER_file_exist); 
                out.writeLong(path.length());
                resume = true;
                fileSize = fileSize-path.length();
                logger.info("Resume at : " + path.length() + 
" Sending "+ fileSize +" bytes more");

            }
        }else
            out.writeUTF("lets go");


        byte[] buffer = new byte[1024];
        // ***********************************
        //  RECEIVE FROM PHONE
        // ***********************************

        int size = 1024;
        int val = 0;

        bos = new BufferedOutputStream(new FileOutputStream(path,resume));

        if(fileSize < size){
            size = (int) fileSize;
        }

        while (fileSize >0) {
            val = in.read(buffer, 0, size);
            bos.write(buffer, 0, val);
            fileSize -= val;
            if (fileSize < size)
                size = (int) fileSize;
        }
        bos.flush();
        bos.close();
        out.writeUTF("file received ok");

        count++;

    }

Answer 1

Found the error and the problem was bad logic from my part. say no more.

I was sending pictures that was being resized just before they where sent.

The problem was when the resume kicked in after a failed transfer
the resized picture was not used, instead the code used the original
pictured that had a larger scale size.

I have now setup a short lived cache that holds the resized temporary pictures.

In the light of the complexity of the app im making I simply forgot that the files during resume was not the same as original.

Answer 2

With a BufferedOutputStream, BufferedInputStream, you need to watch out for the following

1. Create BufferedOutputStream before BuffererdInputStream (on both client and server)
2. And flush just after create.
3. Flush after every write (not just before close)

That worked for me.

Edited

Add sentRequestTime, receivedRequestTime, sentResponseTime, receivedResponseTime to your packet payload. Use System.nanoTime() on these, run your server and client on the same host, use ExecutorService to run multiple clients for that server, and plot your (received-sent) for both request and response packets, time delay on a excel chart (some csv format). Do this before bufferedIOStream and afterIOStream. You will be pleased to know that your performance has boosted by 100%. Made me very happy to plot that graph, took about 45 mins.

I have also heard that using custom buffer's further improves performance.

Edited again In my case I am using Object IOStreams, I have added a payload of 4 long variables to the object, and initialize sentRequestTime when I send the packet from the client, initialize receivedRequestTime when the server receives the response, so and so forth for the response from server to client too. I then find the difference between received and sent time to find out the delay in response and request. Be careful to run this test on localhost. If you run it between different hardware/devices, their actual time difference may interfere with your test results. Since requestReceivedTime is time stamped at the server end and the requestSentTime is time stamped at the client end. In other words, their own local time is stamped (obviously). And both of these devices running the exact same time to the nano second is not possible. If you must run it between different devices atleast make sure that you have ntp running (to keep them time synchronized). That said, you hare comparing the performance before and after bufferedio (you dont really care about the actual time delays right ?), so time drift should not really matter. Comparing a set of results before buffered and after buffered is your actual interest.

Enjoy!!!