Is there a better way to check file size using SCP before downloading?

Question

I have this code to download a file from a remote machine, but I want to limit it to files that are less than 5MB.

So far the code works. but is there a better way to check the filesize before downloading?

Net::SCP.start(hname, uname, :password => pw ) do|scp|
fsize=scp.download!("#{remdir}/#{filname}").size;puts fsize
  scp.download!("#{remdir}/#{filname}",  "#{filname}") do |ch, name, sent, total|
  File.open(lfile, 'a'){ |f| f.puts "#{name}: #{(sent.to_f * 100 / total.to_f).to_i}% complete"}
  #puts "Sending : #{(sent.to_f * 100 / total.to_f).to_i}% complete"
  #puts "#{name}: #{sent}/#{total}"
  #print "\r#{name}: #{(sent.to_f * 100 / total.to_f).to_i}%"
  end
end

Does this cause any problem if I am use it for large files?

fsize=scp.download!("#{remdir}/#{filname}").size;puts fsize

This page says the file will be returned as a string: http://ruby.about.com/od/ssh/ss/netscp_6.htm

Update: I tried SFTP aswell. first it did not work for full path to file. and secondly it did not do what i wanted. so was using scp.download!().size. i know i am doing the download twice :(

require 'net/sftp'
# did not take full path "/user/myname/filename"
remote_path="somefile.txt"
 Net::SFTP.start(hname, uname, :password => pw) do |sftp|
   attrs = sftp.stat!("rome_desc.txt") ; puts attrs  # gave   #   ☼     ↨?%  (?  '→  ??Q{ ?Qt;?
     sftp.stat!(remote_path) do |response| puts response   #returned no such file (2)
 # but did not do below upload operation.
      unless response.ok?
    sftp.upload!("#{filname}", remdir)
  end
end
end

Update:2 Solution found the solution using the the comments provided from below users and after searching the net.

    Net::SFTP.start(hname, uname, :password => pw) do |sftp| #, :verbose => Logger::DEBUG
        sftp.dir.glob("./#{remdir}", "#{filname}") do |entry| p entry.name
        file_size=entry.attributes.size;   file_size = '%.2f' % (("#{file_size}").to_f / 2**20)    ;  File.open(lfile, 'a'){ |f| f.puts "File size is #{file_size} mb"}
            if file_size < file_size_param then
                sftp.download!("#{remdir}/#{filname}", filname)
            else
            File.open(lfile, 'a'){ |f| f.puts "File size is greater than #{file_size_param} mb. so can not Download File"}
            end
        end
    end

used .attributes.size to obtain the file size and perform the download operation by checking the filesize.

 sftp.dir.glob("./#{remdir}", "#{filname}") do |entry| p entry.name
        file_size=entry.attributes.size

Answer 1

Your current code downloads the file, checks the size of the downloaded file (presumably to check if it is less than 5MB, but you don't actually do that), and then downloads it again. Even if you did something with fsize, it's too late to have not downloaded it.

I'd look into the sftp gem rather than scp; it should be pretty straightforward to do what you want with sftp, but not with scp.

Answer 2

Does this cause any problem if I am use it for large files?

We don't know because we don't know how fast your internet connection is, how much RAM you have, how fast the pipe is from the host you're downloading the file from?

Basically though, you are reading the file twice, once into memory to see how big it is, then again if it meets your requirement, which seems really... silly.

You're doubling the traffic to the host you're reading from and on your network connection, and, if the file is larger than RAM on your local machine, it is going to go nuts.

As Darshan says, look at using Net::SFTP. It will give you the ability to query the file's size before you try to load it, without pulling the entire thing down. It's a bit more complicated to use, but that complexity translates into flexibility.

---

"/user/myname/filename"

(S)FTP might not necessarily have its base path where it can see that file. To probe the system and figure out, ask the system, via the SFTP connection, what its current directory is when you first login, then ask it for the files it can see using something like this example from the Net::STFP docs:

sftp.dir.glob("/base/path", "*/**/*.rb") do |entry|
  p entry.name
end

That will recursively look through the "/base/path" hierarchy, searching for all "*.rb" files.