plotting loess with standard errors in R causes integer overflow

Question

I am attempting to use predict with a loess object in R. There are 112406 observations. There is one particular line inside stats:::predLoess which attempts to multiply N*M1 where N=M1=112406. This causes an integer overlow and the function bombs out. The line of code that does this is the following (copied from predLoess source):

L <- .C(R_loess_ise, as.double(y), as.double(x), as.double(x.evaluate[inside, 
]), as.double(weights), as.double(span), as.integer(degree), 
as.integer(nonparametric), as.integer(order.drop.sqr), as.integer(sum.drop.sqr), 
as.double(span * cell), as.integer(D), as.integer(N), as.integer(M1), 
double(M1), L = double(N * M1))$L

Has anyone solved this or found a solution to this problem? I am using R 2.13. The name of this forum is fitting for this problem.

Answer 1

It sounds like you're trying to get predictions for all N=112406 observations. First, do you really need to do this? For example, if you want graphical output, it's faster just to get predictions on a small grid over the range of your data.

If you do need 112406 predictions, you can split your data into subsets (say of size 1000 each) and get predictions on each subset independently. This avoids forming a single gigantic matrix inside predLoess.