python read specific lines from file and continue

Question

I am trying to read specific lines from file and continue reading after ending the process of each chunk. Let's say, I have 19000 lines in a file. Each time, I will extract first 19 lines,make some calculation with those lines and write the output in another file. Then I will extract again the next 19 lines and do the same processing. So, I tried to extract lines in the following way:

n=19
x = defaultdict(list)

i=0

fp = open("file")
for next_n_lines in izip_longest(*[fp] *n):
    lines = next_n_lines

    for i, line in enumerate(lines): 
        do calculation
    write results

Here the code works for first chunk. Could any of you please help me, how can I continue for next n number of chunk ? Thanks a lot in advance!

Answer 1

Your code already extracts lines in groups of 19 lines so I'm not sure what your issue is.

I can clean up your solution slightly, but it does the same thing as your code:

from itertools import izip_longest

# grouping recipe from itertools documentation
def grouper(n, iterable, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx
    args = [iter(iterable)] * n
    return izip_longest(fillvalue=fillvalue, *args)

def process_chunk(chunk):
    "Return sequence of result lines.  Chunk must be iterable."
    for i, line in enumerate(chunk):
        yield 'file-line {1:03d}; chunk-line {0:02d}\n'.format(i, int(line))
    yield '----------------------------\n'

Here is some test code that demonstrates that every line is visited:

from StringIO import StringIO

class CtxStringIO(StringIO):
    def __enter__(self):
        return self
    def __exit__(self, *args):
        return False

infile = CtxStringIO(''.join('{}\n'.format(i) for i in xrange(19*10)))
outfile = CtxStringIO()


# this should be the main loop of your program.
# just replace infile and outfile with real file objects
with infile as ifp, outfile as ofp:
    for chunk in grouper(19, ifp, '\n'):
        ofp.writelines(process_chunk(chunk))

# see what was written to the file
print ofp.getvalue()

This test case should print lines like this:

file-line 000; chunk-line 00
file-line 001; chunk-line 01
file-line 002; chunk-line 02
file-line 003; chunk-line 03
file-line 004; chunk-line 04
...
file-line 016; chunk-line 16
file-line 017; chunk-line 17
file-line 018; chunk-line 18
----------------------------
file-line 019; chunk-line 00
file-line 020; chunk-line 01
file-line 021; chunk-line 02
...
file-line 186; chunk-line 15
file-line 187; chunk-line 16
file-line 188; chunk-line 17
file-line 189; chunk-line 18
----------------------------

Answer 2

This solution needs not loading all lines in memory.

n=19
fp = open("file")
next_n_lines = []
for line in fp:
    next_n_lines.append(line)
    if len(next_n_lines) == n:
        do caculation
        next_n_lines = []
if len(next_n_lines) > 0:
    do caculation
write results

Answer 3

It's not clear in your question, but I guess the calculations you make depend on all the N lines you extract (19 in your example).

So it's better to extract all these lines and then do the work:

N = 19
inFile = open('myFile')
i = 0
lines = list()

for line in inFile:
    lines.append(line)
    i += 1
    if i == N:
        # Do calculations and save on output file
        lines = list()
        i = 0