Efficiently consolidating source code from multiple files to a single file including (Roxygen) comments

Question

Question

What is an efficient way to consolidate source code from multiple source files to a single source file - including Roxygen documentation comments and possibly also including other comments?

I remember there was a parsing method from some package that would account for comments (putting it in some attribute "field"), but I can't find it anymore.

---

Background info

For mainly two reasons I like the flexibility to consolidate source code from a given number of source files to a single source file:

1. In order to keep my head up, I'm sticking to a "one definition per file" paradigm where each source file contains exactly one definition (function, S4 method, S4 Reference Class etc.). Also, these source files might be stored in various subdirectories of my "source directory" and might thus even have the same file name. Yet, in order to put together a true R package, it's sometimes better to group multiple defs to one source file. In cases of duplicated file names I even need to.
2. When parallelizing, it's handy to be able to group all the source code necessary in one file and push that to the worker processes so they can source the code

Homework

Here's my current solution; it feels "okay", but

1. I'm feeling there might be better, that is more efficient, ways to do it
2. It seems a bit fragile with respect to detecting Roxygen code

Creating example source files

foo1 <- function(x) {message("I'm foo #1"); return(TRUE)}
roxy.1 <- c(
    "#' Title foo1()",
    "#'", 
    "#' Description foo1().",
    "##' This line is commented out",
    "#'", 
    "#' @param x Some R object that doesn't matter.",
    "#' @return \\code{TRUE}.",
    "#' @references \\url{http://www.something.com/}",
    "#' @author Janko Thyson \\email{john.doe@@something.com}",
    "#' @seealso \\code{\\link{foo2}}",
    "#' @example inst/examples/foo1.R"
)

foo2 <- function(y) {message("I'm foo #2"); return(FALSE)}
roxy.2 <- c(
    "#' Title foo2()",
    "#'", 
    "#' Description foo2().",
    "##' This line is commented out",
    "#'", 
    "#' @param y Some R object that doesn't matter.",
    "#' @return \\code{FALSE}.",
    "#' @references \\url{http://www.something.com/}",
    "#' @author Janko Thyson \\email{john.doe@@something.com}",
    "#' @seealso \\code{\\link{foo1}}",
    "#' @example inst/examples/foo2.R"
)

dir.create("src/functions", recursive=TRUE, showWarnings=FALSE)
dir.create("src/conso", recursive=TRUE, showWarnings=FALSE)

write(roxy.1, file="src/functions/foo1.R")
write(deparse(foo1), file="src/functions/foo1.R", append=TRUE)
write(roxy.2, file="src/functions/foo2.R")
write(deparse(foo2), file="src/functions/foo2.R", append=TRUE)

Consolidation function

consolidateThis <- function(
    path="src/functions",
    path.conso="src/conso/src_functions.R",
    rgx.roxy="^(#' ?|##' ?)(\\w*|@|$)",
    do.overwrite=TRUE,
    do.roxygen=TRUE,
    ...
) {
    if (!file.exists(path)) {
        stop("Check your 'path' argument")
    }
    files <- list.files(path, full.names=TRUE)
    if (do.overwrite) {
        file.create(path.conso)
    }
    sapply(files, function(ii) {
        this <- readLines(con=ii, warn=FALSE)
        code <- base::parse(text=this)
        if (do.roxygen) {     
            idx.roxy <- grep(rgx.roxy, this)
            if (length(idx.roxy)) {
                if (length(idx.roxy) == 1) {
                    stop("Weird roxygen code (just a one-liner)") 
                }
                bench <- seq(from=idx.roxy[1], max(idx.roxy))
                if (!all(bench %in% idx.roxy)) {
                    stop("Breaks in your roxygen code. Possibly detected comments that aren't roxygen code")
                }
                code.roxy <- this[idx.roxy]
                write(code.roxy, file=path.conso, append=TRUE)
            }
        }
        write(c(deparse(code[[1]]), ""), file=path.conso, append=TRUE)
    })
    return(path.conso)
}

Applying the function

path <- consolidateThis()
> path
[1] "src/conso/src_functions.R"

So now there's a source file 'src/conso/src_functions.R' containing the consolidated code

Answer 1

Do you have a particular need to parse (and then deparse) the functions' source? If not, you can simplify the code quite a bit.

The following produces exactly the same output as ConsolidateThis().

ConsolidateThis2 <-
function(path="src/functions",
         path.conso="src/conso/src_functions.R",
         overwrite = TRUE) {
    if(overwrite) cat("", file = path.conso) # Blank out the file's contents

    ## A function to append infile's contents to outfile and add 2 <RET>          
    prettyappend <- function(infile, outfile) {
        file.append(outfile, infile)
        cat("\n\n", file = outfile, append = TRUE)
    }

    ## Append all files in 'path.conso' to file 'path'
    sapply(dir(path, full.names=TRUE), prettyappend, path.conso)
}

ConsolidateThis2()