Using MemoryMappedFile and FileSystemWatcher to detect new entries to logfile

Question

I have a logfile that is written by a 3rd party application and I'd like my application to "read" that log file in real/near-time, parse the new log entries and act upon certain events.

My thought was that I could achieve this with a combination of FileSystemWatcher (to signal file changes) and MemoryMappedFile (to continue reading from a certain offset).

However, since this is the first time I'm using MemoryMappedFiles I do run into some issues which probably arise from not understanding the concept correctly (e.g. I'm unable to open the existing File as it's in use by the other process).

I was wondering if someone has an example of how to use MemoryMappedFiles to read a file that is locked by another process?

Thanks,

Tom

EDIT:

From the comments, it looks like Memory Mapped Files won't help me accessing files that have an exclusive lock. However, "tail" tools like, e.g. Baretail (http://www.baremetalsoft.com/baretail/index.php) are able to do just that. It has no problem reading the file that has an exclusive lock from another application in 1s intervals). So, there has to be some way to do this?

EDITEDIT:

To answer my own question, the trick in opening a locked file is, creating the FileStream with the following access flags:

fileStream = new System.IO.FileStream(fileName, FileMode.Open, FileAccess.Read, FileShare.Delete | FileShare.ReadWrite);

Answer 1

To answer my own question, the trick in reading a locked file is creating the FileStream with the following access flags:

FileStream fileStream = new System.IO.FileStream(fileName, FileMode.Open, FileAccess.Read, FileShare.Delete | FileShare.ReadWrite);

Now it's just a matter of either doing interval based polling or looking for FileSystemWatcher change events to detect file changes

Answer 2

I'm not sure if MemoryMappedFiles are going to help you. Take a look at FileStream:

var stream = new FileStream(path, FileMode.Open, FileShare.Read);
stream.Seek(offset, SeekOrigin.Begin);

Though if the 3rd-party application has the file locked exclusively, there's not much that you can do about it...

Answer 3

[Begin 2nd EDIT]

One more idea...

If the third party app happens to use a logging framework like NLog, log4net, or System.Diagnostics, you could still write your own Target/Appender/TraceListener and route the messages somewhere that you could look at them (such as a file that is not opened exclusively, to another process, etc).

If your third party app is using a logging framework, we probably would have heard about it by now ;-)

[End 2nd EDIT]

[Begin EDIT]

I think I misread the question. It sounded at first like you were using a third party library that had logging implemented and you wanted to do this parsing from within the program that was generating the logging. Having reread your question, it sounds like you want to "listen" to the log file from outside of the application. If that is the case, my answer probably won't help you. Sorry.

[End EDIT]

I don't have anything to offer about MemoryMappedFiles, but I wonder if you could achieve what you are after by writing a custom listener/target/appender for the 3rd party logging system?

For example, if you are using NLog, you could write a custom Target and direct all of your logging messages there (while also directing them to the "real" Target(s)). This way you get crack at each log message as it is logged (so it is actually real time, not near real time). You could do the same thing with log4net and System.Diagnostics.

Note that NLog even has a "MethodCall" target. To use that one you only have to write a static method with the correct signature. I don't know if log4net has a similar concept to this.

This seems like it would be easier to get working reliably than trying to read and parse the log file as it is being written by the third party software.

Answer 4

If the file is "in use", there isn't anything that can be done about that. It truly is "in use". MemoryMappedFiles are for either reading large amounts of data off the drive or sharing data with other programs. It will not help getting around the "in use" limitation.

Answer 5

Memorymapped files are under the same restrictions as the FileStream you initialize it with, be sure that you initialize your Memory-Mapped-File like this

var readerStream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);

var mmf = MemoryMappedFile.CreateFromFile(readerStream, null, 0, MemoryMappedFileAccess.Read, null, HandleInheritability.None, false);

If some other process have completely locked it even from writing you're in bad luck, not sure if there's a way around that. Perhaps use some timer do detect when the process has stopped writing to it.

Answer 6

I've done something similar just for monitoring log files on a console (as opposed to processing), but the principles are the same. Like you, I use a FileSystemWatcher, and the important logic is in my OnChanged event handler:

case WatcherChangeTypes.Changed:
{
    System.IO.FileInfo fi = new FileInfo(e.FullPath);

    long prevLength;

    if (lastPositions.TryGetValue(e.FullPath, out prevLength))
    {
        using (System.IO.FileStream fs = new FileStream(
           e.FullPath, FileMode.Open, FileAccess.Read))
        {
            fs.Seek(prevLength, SeekOrigin.Begin);
            DumpNewData(fs, (int)(fi.Length - prevLength));
            lastPositions[e.FullPath] = fs.Position;
        }
    }
    else
      lastPositions.Add(e.FullPath, fi.Length);

    break;
}

where lastPositions is

Dictionary<string, Int64> lastPositions = new Dictionary<string, long>();

and DumpNewData is simply

private static void DumpNewData(FileStream fs, int bytesToRead)
{
    byte[] bytesRead = new byte[bytesToRead];
    fs.Read(bytesRead, 0, bytesToRead);
    string s = System.Text.ASCIIEncoding.ASCII.GetString(bytesRead);
    Console.Write(s);
}