file offsets on windows

Question

Is there an easy way, preferably with a scripting language or a small tool that could be called through a batch file, to operate on a text file, mark an offset, and put everything after the offset into a new file?

I have a text file added to nightly, and I would like to make it so that the end of the file is marked, then after new data is added, only data between the offset and the end is processed. I can not do this with just strings or delimiters as it is blob data.

edit: The text file is created by running a ms access macro from a scheduled tasks, which exports the data as a csv file. In considering Patricks suggestion, I would like to know if it would be possible to add a wildcard such as the date to the filename, to always have a different file. This file will then be scp'd to a linux server, where it will be loaded into a mysql database.

Answer 1

Assuming you're currently exporting the data from the Access database with a script already:

@echo OFF

:: Force a new line and add a marker; assuming your file is data.txt.
@echo. >> data.txt
@echo **MARKER** >> data.txt

:: Run your export here: these lines just simulate the export.
@echo Test Line 1 >> data.txt
@echo Test Line 2 >> data.txt

:: Find line number of last marker:
for /f "usebackq delims=:" %%I in (`findstr /N "**MARKER**" data.txt`) do (
    set LAST_MARKER=%%I
)

:: Get all the lines after the last marker
for /f "skip=%LAST_MARKER% tokens=*" %%L in (data.txt) do (
    @echo %%L >> new_data.txt
)

The output in new_data.txt will be:

Test Line 1
Test Line 2

Answer 2

It is simple with python:

import sys

def divide_file(fname, mark):
    mark_found = 0
    f = file(fname, 'r')
    for line in f.readlines():
        if mark in line:
            mark_found = 1
        if mark_found:
            print line.rstrip()
    f.close()

divide_file(sys.argv[1], sys.argv[2])

Usage & output example:

c:\tmp>divide_file.py divide_file.py close
        f.close()

divide_file(sys.argv[1], sys.argv[2])

Answer 3

I could think of tail, bash and other utilities from Unix-like systems. You could get those on Windows by minimally installing MSYS. Documentation and examples referring to these utilities are quite easy to find. And bash stuff is way stronger than Windows batch files. The script would look something like this:

#!/bin/bash

PREV_SIZE=`du -b text_file`
write_something_to_file text_file
CURR_SIZE=`du -b text_file`
let NUM=$PREV_SIZE-$CURR_SIZE
tail -c $NUM > new_text_file