Find file with the most update information

Question

I have a list of log files, and I need to find which one has a latest edition of a specific line, and all or none could have this line.

The lines in the files look like this:

2013/01/06 16:01:00:283  INFO ag.doLog: xxxx xxxx xxxx xxxx

And I need a line lets say

xx/xx/xx xx:xx:xx:xxx  INFO ag.doLog: the line i need

I know how to get an array of files, and if I scan backwards I could find the latest latest line in each file (if it exists).

Biggest problem is that the file could be big (2k lines?) and I want to find the line in a relative fast way (a few seconds), so I am open for suggestion.

Personal ideas: If a file has the line at X time, then any file that has not found the line before X time should not be scan anymore. This will require to search all files at the same time, which i dont know how.

Atm the code breaks, and I suppose if lack of memory.

Code:

if(files.length>0)  {  //in case no log files exist
    System.out.println("files.length: " + files.length);
    for(int i = 0; i < files.length; i++)  {  ///for each log file look for string
        System.out.println("Reading file: " + i + " " + files[i].getName());
        RandomAccessFile raf = new RandomAccessFile(files[i].getAbsoluteFile(), "r"); //open log file
        long lastSegment = raf.length(); //Finds how long is the files
        lastSegment = raf.length()-5;    //Sets a point to start looking
        String leido = "";
        byte array[] = new byte[1024];    
        /*
        * Going back until we find line or file is empty.
        */
        while(!leido.contains(lineToSearch)||lastSegment>0)  {
            System.out.println("leido: " + leido);
            raf.seek(lastSegment);           //move the to that point
            raf.read(array);                 //Reads 1024 bytes and saves in array
            leido = new String(array);       //Saves what is read as a string
            lastSegment = lastSegment-15;    //move the point a little further back
        }
        if(lastSegment<0)   {
           raf.seek(leido.indexOf(lineToSearch) - 23); //to make sure we get the date (23 characters long) NOTE: it wont be negative. 
           raf.read(array);                 //Reads 1024 bytes and saves in array
           leido = new String(array);       //make the array into a string
           Date date = new SimpleDateFormat("MMMM d, yyyy", Locale.ENGLISH).parse(leido.substring(0, leido.indexOf(" INFO "))); //get only the date part
           System.out.println(date); 
           //if date is bigger than the other save file name
        }
     }
}

Answer 1

I find the code difficult to verify. One could split the task in a backwards reader, which reads lines from file end to start. And use that for parsing dates line wise.

Mind, I am not going for nice code, but something like this:

public class BackwardsReader implements Closeable {

    private static final int BUFFER_SIZE = 4096;

    private String charset;
    private RandomAccessFile raf;
    private long position;
    private int readIndex;
    private byte[] buffer = new byte[BUFFER_SIZE];

    /**
     * @param file a text file.
     * @param charset with bytes '\r' and '\n' (no wide chars).
     */
    public BackwardsReader(File file, String charset) throws IOException {
        this.charset = charset;
        raf = new RandomAccessFile(file, "r");
        position = raf.length();
    }

    public String readLine() throws IOException {
        if (position + readIndex == 0) {
            raf.close();
            raf = null;
            return null;
        }

        String line = "";
        for (;;) { // Loop adding blocks without newline '\n'.

            // Search line start:

            boolean lineStartFound = false;
            int lineStartIndex = readIndex;
            while (lineStartIndex > 0) {
                if (buffer[lineStartIndex - 1] == (byte)'\n') {
                    lineStartFound = true;
                    break;
                }
                --lineStartIndex;
            }
            String line2;
            try {
                line2 = new String(buffer, lineStartIndex, readIndex - lineStartIndex,
                        charset).replaceFirst("\r?\n?", "");
                readIndex = lineStartIndex;
            } catch (UnsupportedEncodingException ex) {
                Logger.getLogger(BackwardsReader.class.getName())
                        .log(Level.SEVERE, null, ex);
                return null;
            }
            line = line2 + line;
            if (lineStartFound) {
                --readIndex;
                break;
            }

            // Read a prior block:

            int toRead = BUFFER_SIZE;
            if (position - toRead < 0) {
                toRead = (int) position;
            }
            if (toRead == 0) {
                break;
            }
            position -= toRead;
            raf.seek(position);
            raf.readFully(buffer, 0, toRead);
            readIndex = toRead;
            if (buffer[readIndex - 1] == (byte)'\r') {
                --readIndex;
            }
        }
        return line;
    }

    @Override
    public void close() throws IOException {
        if (raf != null) {
            raf.close();
        }
    }
}

And a usage example:

public static void main(String[] args) {
    try {
        File file = new File(args[0]);
        BackwardsReader reader = new BackwardsReader(file, "UTF-8");
        int lineCount = 0;
        for (;;) {
            String line = reader.readLine();
            if (line == null) {
                break;
            }
            ++lineCount;
            System.out.println(line);
        }
        reader.close();
        System.out.println("Lines: " + lineCount);
    } catch (IOException ex) {
        Logger.getLogger(App.class.getName()).log(Level.SEVERE, null, ex);
    }
}