skipping first half of a 59GB fastq file to process last half: read line-by-line, or fgetpos?

Question

I have 2 ~59GB text files in ".fastq" format. fastq files are genomics read files from a sequencer. Every 4 lines is a new read, but the lines are of variable size.

The filesize is roughly 59GB, and there are about 211M reads-- which means, give or take, approximatley 211M*4 = 844M lines. The program I'm using, Bowtie, currently has the ability to do the following options:

"--skip 105M --qupto 105M"

which essentially means "skip the first 105M reads and only process up to the next 105M reads." In this way you can break up processing of the file. The problem is, the way that it does the skipping is incredibly slow. It just reads the first 105M reads as it normally would, but doesn't process them. Then it starts comparisons once it gets to the read value it was given.

I am wondering if I can use something like C/C++'s fsetpos to set the position to the middle of the file [or wherever] which I realize will probably put me somewhere in the middle of a line, and then from there find the beginning of the first full read to start processing rather than waiting for it to read approximately 422M lines until it gets where it needs to go. Does anybody have experience doing fsetpos on such a large file, and know whether or not the performance is any better than it is how it's currently doing it?

Thanks-- Nick

Answer 1

Yes, you can position to the middle of a file using C++.

For huge files, the performance is usually better than reading the data.

In general, the process for positioning within a file:

1. A request is made to read the directory entry for the file.
2. The directory is searched to find the track and sector for the file position.
3. Note: Some filesystems may have directory extensions for large files, thus more data will need to be read.
4. On the next read, the hard drive is told to go to the given track and sector, then read in data.

You are saving time from all the previous data to pass through the communications port and into memory (or ignored).