How can have SQL Server tail a log file and write new rows to a table?

Question

I have an application that creates a text log file and I have a daily SQL Server job that truncates a log table and uses bulk insert to load the current days logs. The log file is written to by many threads and it's hard to read without manually sorting out the lines by thread.

Is there a way to have SQL Server 'tail' a given file and keep the table up to date without waiting for the end-of-day batch? I'd rather not run the batch multiple times during the day as this table will be monitored/searched/displayed by a web service. Logging directly to the table from the application is not currently an option either.

Answer 1

I would have done it through CLR and the job.

1. To have a job that runs at a certain intervals.
2. Create CLR function that accepts file name and current line number in the file.

CLR will open file and based on a give number will fetch data after specified line and return back to sql server.

Line number will be stored in auxiliary table for the future synchronization purposes.

Here is article how to write CLR to return table back to sql server : http://msdn.microsoft.com/en-us/library/ms131103.aspx

Good luck.