Very large fseek forward equivalent for stdin?

Question

I have a very large, known number of bytes on stdin, and wish to discard a large (also known) number of them before reading the portion of interest (in other words, I want to fseek forward by a large integer, but fseek isn't defined for pipes). The simplest way to achieve this seems to be a large number of calls to fgetc, and the first alternative is to use a single call to fread with a large scratch pointer allocated to store the result. The first is very slow, and the second uses a potentially unbounded amount of memory for no good reason. Making multiple smaller reads solves the unbounded memory usage issue, but introduces a free parameter (the chunk size) which probably has a different fastest value for every machine and OS combination.

Are there any alternatives that achieve this goal in a neat, efficient fashion? POSIX is assumed.

Answer 1

There is no way to "skip" data on a pipe - you have to read it.

If it's a very large block, you will want to use a medium-sized buffer (as a compromise between overhead and memory usage), something like this:

 size_t dataToRead = some_large_number;

 while(dataToRead)
 {
    char buffer[4096];
    size_t toread = min(sizeof(buffer), dataToRead);
    size_t nread = fread(buffer, 1, toread, stdin);
    dataToRead -= nread;
 }

The size, 4096 is a rather arbitrary choice - but it's big enough not to cause a HUGE number of reads to the input, and small enough to not use crazy amounts of stack-space. It is unlikely that you will gain/loose much from changing this size.