How can I save a file to /tmp/ via ftp on cedar

Question

I have a rake task that connects to an ftp server, downloads a file, and then gunzips it. This works locally, but when I run the task on my heroku cedar server I get no feed back after executing net::ftp.getbinaryfile.

Here is the code to download the file:

tempfile = "#{Rails.root}/tmp/#{Process.pid}_#{MyApp::Application.config.products_bbcom_file}"
      ftp = Net::FTP.new()
      puts "connecting to ftp server #{MyApp::Application.config.products_bbcom_host}"
      ftp.connect(MyApp::Application.config.products_bbcom_host)
      puts "logging in"
      ftp.login(MyApp::Application.config.products_bbcom_user,MyApp::Application.config.products_bbcom_pass)
      puts "changing directory"
      files = ftp.chdir(MyApp::Application.config.products_bbcom_path)
      #files = ftp.list('n*')
      puts "downloading file #{MyApp::Application.config.products_bbcom_file} to #{tempfile}"
      ftp.getbinaryfile(MyApp::Application.config.products_bbcom_file, tempfile, 1024)
      ftp.close      

When I execute the task on heroku: heroku run rake db:import and tail the logs I see:

2012-09-10T16:26:24+00:00 heroku[run.1]: Awaiting client
2012-09-10T16:26:24+00:00 heroku[run.1]: Starting process with command `bundle exec rake db:import`
2012-09-10T16:26:25+00:00 heroku[run.1]: State changed from starting to up

And the out put from the task gets to:

== Starting ==
Connecting to database specified by DATABASE_URL
connecting to ftp server datatransfer.cj.com
logging in
changing directory
downloading file

The script gets to the point where it is trying to download the file to #{Rails.root}/tmp/ but then never responds again. This only takes a couple seconds locally, but I waited a few minutes and the task doesn't do anything.

From the heroku dev site it seems like you can save files to #{Rails.root}/tmp/ on cedar. Is this possible? If so am I taking the wrong approach?

Answer 1

It is possible to download a file vit NET::FTP, but heroku only supports PASSIVE mode FTP. I needed to specify that I wanted to use passive mode. Here is a simplified example:

ftp = Net::FTP.new()
ftp.passive = true
ftp.connect(host)
ftp.login(username,password)
files = ftp.chdir(path_to_file)
ftp.getbinaryfile(filename, tempfile, 1024)
ftp.close

Answer 2

You can save files to ./tmp, but know that if the dyno gets restarted that data will be gone. Additionally, one dyno cannot access another dyno's tmp directory.

For data processing and short-term storage, the tmp directory is a great spot.

Answer 3

Are you trying to backup/import your database?

Use PGBACKUPS. The dump goes onto S3. You can download or import from there.

https://devcenter.heroku.com/articles/pgbackups#restoring-from-a-backup

I used to do the same thing you are via FTP before switching to heroku. pgbackups makes it so much easier.