Delphi7 - How can i copy a file that is being written to

Question

I have an application that logs information to a daily text file every second on a master PC. A Slave PC on the network using the same application would like to copy this text file to its local drive. I can see there is going to be file access issues.

These files should be no larger than 30-40MB each. the network will be 100MB ethernet. I can see there is potential for the copying process to take longer than 1 second meaning the logging PC will need to open the file for writing while it is being read.

What is the best method for the file writing(logging) and file copying procedures? I know there is the standard Windows CopyFile() procedure, however this has given me file access problems. There is also TFileStream using the fmShareDenyNone flag, but this also very occasionally gives me an access problem too (like 1 per week).

What is this the best way of accomplishing this task?

My current File Logging:

procedure FSWriteline(Filename,Header,s : String);
var LogFile : TFileStream;
line : String;
begin
     if not FileExists(filename) then
     begin
          LogFile := TFileStream.Create(FileName, fmCreate or fmShareDenyNone);
          try
             LogFile.Seek(0,soFromEnd);
             line := Header + #13#10;
             LogFile.Write(line[1],Length(line));
             line := s + #13#10;
             LogFile.Write(line[1],Length(line));
          finally
                 logfile.Free;
          end;
     end else begin
         line := s + #13#10;
         Logfile:=tfilestream.Create(Filename,fmOpenWrite or fmShareDenyNone);
         try
            logfile.Seek(0,soFromEnd);
            Logfile.Write(line[1], length(line));
         finally
            Logfile.free;
         end;
     end;
end;

My file copy procedure:

procedure DoCopy(infile, Outfile : String);
begin
     ForceDirectories(ExtractFilePath(outfile)); //ensure folder exists
     if FileAge(inFile) = FileAge(OutFile) then Exit; //they are the same modified time
     try
        { Open existing destination }
        fo := TFileStream.Create(Outfile, fmOpenReadWrite or fmShareDenyNone);
        fo.Position := 0;
     except
           { otherwise Create destination }
           fo := TFileStream.Create(OutFile, fmCreate or fmShareDenyNone);
     end;
     try
        { open source }
        fi := TFileStream.Create(InFile, fmOpenRead or fmShareDenyNone);
        try
           cnt:= 0;
           fi.Position := cnt;
           max := fi.Size;
           {start copying }
           Repeat
                 dod := BLOCKSIZE; // Block size
                 if cnt+dod>max then dod := max-cnt;
                 if dod>0 then did := fo.CopyFrom(fi, dod);
                 cnt:=cnt+did;
                 Percent := Round(Cnt/Max*100);
           until (dod=0)
        finally
               fi.free;
        end;
     finally
            fo.free;
     end;
end;

Answer 1

I would suggest not closing and reopening the shared file over and over to begin with. Since you write to it every second, that is just needless overhead.

On the Master side, create and close the file (the fmCreate flag cannot be used with other flags!), then re-open it in fmOpenWrite mode with fmShareDenyWrite sharing, leave it open, and write to it when needed.

On the Slave side, open the file in fmOpenRead mode with fmShareDenyNone sharing, leave it open, and read from it every second. No need to copy the entire shared file over the network every time. That is wasted bandwidth. Just read whatever new data has been written in the past few seconds and that is all. If the Slave needs the data to be stored in a local file, then it can manage a separate local file independantly of the shared file, pushing new data into the local file when needed.

Answer 2

To deal with your specific occasional recurring problem:

You don't say what version of Delphi you are using.

There is a bug in the TFileStream.Create() constructor up to and including version 2007 (at least). This might explain your occasional concurrency problems.

Having said that, I believe the bug is more likely to result in files not being created as expected (when a ShareMode is additionally specified), tho this may then in turn lead to your concurrency problem.

One way around this might be when the file needs to be created, first create the file then simply open it for writing as a separate constructor call - this actually makes file creation a separate step, with file writing a consistent part of the process:

  if not FileExists(filename) then
  begin
    // There may be a more efficient way of creating an empty file, but this 
    //  illustrates the approach

    LogFile := TFileStream.Create(FileName, fmCreate);
    LogFile.Free;

    line := Header + #13#10 + s + #13#10;
  end
  else
    line := s + #13#10;

  Logfile:=tfilestream.Create(Filename,fmOpenWrite or fmShareDenyNone);
  try
    logfile.Seek(0,soFromEnd);
    Logfile.Write(line[1], length(line));
  finally
    Logfile.free;
  end;

Answer 3

Use the standard Append file creation/opening command, use write to update the log, and close the file immediately.

Use a job on the operating system to copy/move the files; have it retry and to launch at a frequency greater than what you require.

If you want to do it from within Delphi then use MoveFile to move the whole thing.

You might want to wrap both the log writes and the moves in try-except so they can be retried a reasonable number of times if the file system (NTFS on Windows?) doesn't resolve the concurrency for you. In the worst case, either:

1. The file got moved and it gets recreated and written to.
2. The file is not moved right away because it is being written to.

If the OS doesn't resolve the race condition, then you will have to give priority to the starved action using a semaphore/lock.

Answer 4

search for a function called "IsFileInUse" or something similar, I'm sure you can use that like:

// master
while IsFileInUse(*AFileName*) do
  Sleep(10);
write-content-to-file

// slave
while IsFileInUse(*AFileName*) do
  Sleep(10);
copy-file-to-a-special-location

and presto!! you're done!!