Applying counter on log

Question

Code:

@echo off

for /f "delims=" %%a in ('wmic OS Get localdatetime  ^| find "."') do set "dt=%%a"

set "YY=%dt:~2,2%"
set "MM=%dt:~4,2%"
set "DD=%dt:~6,2%"

set "today=%YY%%MM%%DD%"

findstr /i /c:"The_database_has_been_locked." "C:\admin\logs\*%today%.log" >> "C:\records\Error.txt"

Problem Description:

I have some log files under path C:\admin\logs\ which are being created everyday having date of that day in their names.

I want to search a specific error message in all the files created today only And as soon as error message found, immediately append the complete line having error message into a text file named Error.txt.

I've scheduled this script to run everyday on every 10 mins.

The problem I am facing here is, Its appending specified error message occured on a particular time instance multiple times . I mean on every run its appending previously parsed error messages too, which I don't want.

I want to appened the error message only one time into Error.txt for a particular time instance of error-occurence on current date.

Could you please help me out here ?

EDIT:Log snippet

Log snippet:

2013-12-06 16:29:31 10.35.44.195 GET /perform/all/per_cubes.asp - 80 - 172.23.45.72 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+Trident/4.0;+NGD_build;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+.NET4.0C;+.NET4.0E;+InfoPath.3;+MS-RTC+LM+8) 200 0 0 328
2013-12-06 16:29:33 10.35.44.195 GET /perform/all/per_cubes.asp |62|80004005|[Microsoft][ODBC_Microsoft_Access_Driver]_The_database_has_been_locked. 80 - 62.239.225.222 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/5.0;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.30729;+.NET+CLR+3.5.30729;+.NET4.0C;+.NET4.0E;+SLCC2) 500 0 0 468
2013-12-06 16:29:44 10.35.44.195 GET /perform/all/per_cubes.asp |62|80004005|Unspecified_error 80 - 10.225.206.76 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+Trident/4.0;+NGD_build;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+.NET4.0C;+.NET4.0E;+InfoPath.3;+MS-RTC+LM+8) 500 0 0 93

Appending error message to Error.txt like below (only once for a specific time on current date):

2013-12-06 16:29:33 10.35.44.195 GET /perform/all/per_cubes.asp |62|80004005|[Microsoft][ODBC_Microsoft_Access_Driver]_The_database_has_been_locked. 80 - 62.239.225.222 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/5.0;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.30729;+.NET+CLR+3.5.30729;+.NET4.0C;+.NET4.0E;+SLCC2) 500 0 0 468

Answer 1

@ECHO OFF
SETLOCAL
SET "sourcedir=c:\sourcedir"
SET "destdir=c:\destdir"

:: Need to put YOUR setting of %TODAY% in here - I'm using constants for testing

SET today=20140109

FOR %%a IN ("%sourcedir%"\beenprocessed*.txt) DO IF /i NOT "%%a"=="%sourcedir%\beenprocessed%today%.txt" DEL "%%a"
IF NOT EXIST "%sourcedir%\*%today%.log" GOTO :eof

ECHO dummy>>"%sourcedir%\beenprocessed%today%.txt"
findstr /i /c:"The_database_has_been_locked." "%sourcedir%\*%today%.log" >"%sourcedir%\beINGprocessednow.txt"
FINDSTR /i /v /b /e /g:"%sourcedir%\beenprocessed%today%.txt" "%sourcedir%\beINGprocessednow.txt" >>"%destdir%\Error.txt"
type "%sourcedir%\beINGprocessednow.txt" >> "%sourcedir%\beenprocessed%today%.txt" 
GOTO :EOF

You'd need to insert your setting of %today% - I just used a constant. Also, I've set source and destination directories as variables. Just a matter of making changes.

I'm not fussed whether the ...beenprocessed... and ...beingprocessednow... files are in %sourcedir%, %destdir%, %temp% or %someotherdir%. Up to the programmer to choose.

The principle is this:

append a dummy string into beenprocessed.... This either creates a file or appends to it, a string that simply won't be of interest (it fails the '...database_has_been...` filter).

Find all of the target records in the logs and place them in a file beingprocessednow.

Find all of the lines in beingprocessednow that don't appear in beenprocessed - ie. the new lines, and append them to the error.txt file.

Append the entire contents of beingprocessednow to beenprocessed which includes the new data

At startup, the beenprocessed files for any day other than today are deleted.

dummy needs to be added to the "words to exclude" because findstr ../g requires at least one string in the file. dummy can't be a valid word which occupies exectly one line (/b /e) because all lines of interest contain the target string.

Answer 2

Presumably the log entries are all unique in some way.

setLocal enableDelayedExpansion
set searchString=The_database_has_been_locked.
set sourceFile=C:\admin\logs\*%today%.log

for /f "delims=~!" %%a in ('findstr /i /c:"%searchString%" "%sourceFile%"') do (
    set skip=0
    for /f "delims=~!" %%b in ('findstr /i /c:"%%a" "Error.txt"') do set skip=1
    if !skip!==0 echo %%a & echo %%a >> Error.txt
)

The internal for loop will only be able to set the variable skip if it finds %%a in Error.txt. Therefore, if skip is not defined then there will not be a duplicate entry, and you can append the error to Error.txt.