perfomance libssh2 sftp upload multiple files

Question

I try to upload multiple files via the libssh2 sftp subsystem to a remote server. My problem is that I have to close the sftp handle and the sftp session always after transmitting one file, otherwise the function libssh2_sftp_open() will fail with "FXP_STATUS 4, unable to open / create file". Because of this reinit of the session and file I lose valuable seconds.

Is there a way to upload multiple files in a loop without reinit of the session and/or handle? I hope this code snippet will show you my problem, here with reinit after one loop gothrough which works, but needs like 4 seconds for uploading 4 kbytes:

 // Transfer at most MAXTRANSFER files
    for (i=0;(i < ((MAXTRANSFER<nfiles)?MAXTRANSFER:nfiles)) && (iRet == SSHH_OK); i++) {
        localfile = malloc(sizeof(char) * (strlen(directory) + strlen(fnames[i]) + 2));
        strcpy(localfile, directory);
        strcat(localfile, "/");
        strcat(localfile, fnames[i]);

        local = fopen(localfile, "rb");
        if (!local) {
            dbgPrintFormat(DEBUG_WARNING, "dir_sftp: Can't open local file %s", localfile);
            iRet = SSHH_ERR_SETTINGS;
        }

        // Init SFTP
        if (iRet == SSHH_OK) {
            sftp_session = libssh2_sftp_init(session);
            if (!sftp_session) {
                dbgPrintFormat(DEBUG_ERROR, "dir_sftp: Unable to init SFTP session (%d)", sftp_session);
                iRet = SSHH_ERR_SETTINGS;
            }
        }

        if (iRet == SSHH_OK) {
            stat(localfile, &fileinfo);
            char *dstfile = malloc(sizeof(char) * (strlen(destdir) + strlen(fnames[i]) + 2));
            strcpy(dstfile, destdir);
            strcat(dstfile, "/");
            strcat(dstfile, fnames[i]);

            dbgPrintFormat(DEBUG_FINE, "dir_sftp: Start sending file %s", localfile);

            // Request a file via SFTP
            sftp_handle = libssh2_sftp_open(sftp_session, dstfile,
                             LIBSSH2_FXF_WRITE|LIBSSH2_FXF_CREAT|LIBSSH2_FXF_TRUNC,
                             LIBSSH2_SFTP_S_IRWXU|LIBSSH2_SFTP_S_IRGRP|LIBSSH2_SFTP_S_IXGRP);

            if (!sftp_handle) {
                dbgPrintFormat(DEBUG_ERROR, "dir_sftp: Unable to open file with SFTP");
                iRet = SSHH_ERR_SETTINGS;
            }

            if (iRet == SSHH_OK) {
                dbgPrintFormat(DEBUG_FINE, "dir_sftp: SFTP session waiting to send file %s",localfile);
                do {
                    nread = fread(mem, 1, sizeof(mem), local);
                    if (nread <= 0) {
                       /* end of file */
                       break;
                    }
                    ptr = mem;

                    do {
                        /* write the same data over and over, until EOF */
                        rc = libssh2_sftp_write(sftp_handle, ptr, nread);
                        // EOF
                        if(rc < 0)
                            break;
                        ptr += rc;
                        nread -= rc;
                    } while (nread);

                } while (rc > 0);
            }

            ulSftpTxTryCount[iSftpTxTryCountIdx]++;

            if (iRet == SSHH_OK) {
                // Remove localfile *** if no error
                if (remove(localfile) == -1) {
                    dbgPrintFormat(DEBUG_WARNING,"dir_sftp: Error removing file: %s", localfile);
                }
                if( successCnt ) {
                    (*successCnt)++;
                }
                dbgPrintFormat(DEBUG_FINE, "dir_sftp: File sent after %lu tries", ulSftpTxTryCount[iSftpTxTryCountIdx]);
                iSftpTxTryCountIdx = (iSftpTxTryCountIdx + 1) % SCP_TRYCNT_NOF;
                ulSftpTxTryCount[iSftpTxTryCountIdx] = 0;
            }

            free (dstfile);


        }
        if (local)
            fclose(local);
        free (localfile);

        // Close file sftp handle
        if (libssh2_sftp_close(sftp_handle) < 0) {
            dbgPrintFormat(DEBUG_WARNING, "dir_sftp: Error closing SFTP handle ");
        }
        // Close sftp session
        libssh2_sftp_shutdown(sftp_session);
    }

Answer 1

You have several problems. First problem (the one you are asking about) is that the server doesn't seem to like more than one file opened by the client at the same time. I.e. this can be a server-specific problem (as well as the issue of libssh2).

The bigger problem is that your approach of sending a block, waiting for result then sending another block is very slow. Third-party SFTP libraries implement what we call pipelining, when several blocks are sent asynchronously in parallel. This significantly speeds up file transfer.

My suggestion is to find some third-party library which was written specifically for file transfer. This will give you much better speed.

Answer 2

The following are items to check when trying to maximize SFTP transfer speeds:

Is there a firewall or network device inspecting or throttling SSH2 traffic on your network? That might be slowing things down. Check you firewall settings. We've had users report solving extremely slow SFTP file transfers by modifying their firewall rules. The SFTP client you use can make a big difference. Try several different SFTP clients and see if you get different results. Network latency will drastically affect SFTP. If the link you are on has a high degree of latency then that is going to be a problem for fast transfers. How powerful is the server machine? Encryption with SFTP is very intensive. Make sure you have a sufficiently powerful machine that isn't being overtaxed during SFTP file transfers (high CPU utilization).