Not creating a copy with ffload

Question

I have a question about a function attribut from the ff package. I have an ff object in a specific directory I want to load. This is what I am using:

setwd(~/Documents/Data)
ffload("Object", overwrite=TRUE)

With OVERWRITE=TRUE, a copy of the object will be saved in a tmp folder and the object will be load from this directory:

[1] "tmp/Rtmpsj1b2f/clone11876c2c0949.ff"

But if you don't want a copy of your object you need to put FALSE to overwrite. If it is the first time you load your ff object it will create a copy in a tmp folder even if overwrite=FALSE! Then I deleted the copy, and re-loaded the object for the second time, I had the following message and no copy:

Warning :
In FUN("Obs_zone_212_19700801_19750731"[[1L]], ...) :
did not overwrite object 'Obs_zone_212_19700801_19750731'

The problem is that I am working with a lot of big files and I don't want to create copies! What can I do ?

Answer 1

Maybe it is good to give an explanation about ffload for your case

You have 2 files in your ~/Documents/Data directory. One called Object and another one called Object.ffData. Object.ffData contains the ff files with the raw data. Object contains an R workspace with the virtual attributes of the ff files (like levels e.g., pointers to the ff files, ...) which is only a few Kb depending on your data structure.

ffload unzips the ff files which are contained in Object.ffData into the folder where you have created the ff files (see getOption("fftempdir") - for your case this was in the tmp/Rtmpsj1b2f folder. If you specify the rootpath argument to ffload, you can unzip the ff files to another directory. As in ffload("Object", overwrite=TRUE, rootpath = "/home/myname/myfolder"). The overwrite argument of ffload indicates that if that ff file which you are unzipping already existed in that directory, it will overwrite it. So far the ff part.

When loading with ffload your file, you are also loading the virtual part in your R session. This is for your case an object which you have named Obs_zone_212_19700801_19750731 (see ls()). The warning you see is that when ffloading your new data, it did not overwrite this object. So you need to rm(Obs_zone_212_19700801_19750731) before ffloading to get rid of the warning.

Hope this helps. If your data are ffdf objects, you can also look at save.ffdf from package ffbase. This does not zip your data but keeps it straight on disk.