I was able to figure out myself the reasons of the issues described above. It took me a lot of time and effort, but it was worth it - now I understand R lists
and lapply()
better.
Essentially, I made three major changes:
1) added textConnection()
and readLines()
to process CSV-like files:
conn <- gzcon(bzfile(file, open = "r"))
tConn <- textConnection(readLines(conn))
However, I've discovered some issues with this approach - see my other SO question: Extremely slow R code and hanging.
2) used correct subscription notation to refer to the appropriate elements of list inside of function(i)
passed to lapply()
:
url <- links[[1]][i]
3) used correct subscription notation to refer to whole list for lapply()
:
data <- lapply(seq_along(links[[1]]), getData)
Thanks to all who participated in and helped answering this question!