Copy a file without using the windows file cache

Question

Anybody know of a way to copy a file from path A to path B and suppressing the Windows file system cache?
Typical use is copying a large file from a USB drive, or server to your local machine. Windows seems to swap everything out if the file is really big, e.g. 2GiB. Prefer example in C#, but I'm guessing this would be a Win32 call of some sort if possible.

Answer 1

Even more important, there are FILE_FLAG_WRITE_THROUGH and FILE_FLAG_NO_BUFFERING.

MSDN has a nice article on them both: http://support.microsoft.com/kb/99794

Answer 2

In C# I have found something like this to work, this can be changed to copy directly to destination file:

    public static byte[] ReadAllBytesUnbuffered(string filePath)
    {
        const FileOptions FileFlagNoBuffering = (FileOptions)0x20000000;
        var fileInfo = new FileInfo(filePath);
        long fileLength = fileInfo.Length;
        int bufferSize = (int)Math.Min(fileLength, int.MaxValue / 2);
        bufferSize += ((bufferSize + 1023) & ~1023) - bufferSize;
        using (var stream = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.None,
                                           bufferSize, FileFlagNoBuffering | FileOptions.SequentialScan))
        {
            long length = stream.Length;
            if (length > 0x7fffffffL)
            {
                throw new IOException("File too long over 2GB");
            }
            int offset = 0;
            int count = (int)length;
            var buffer = new byte[count];
            while (count > 0)
            {
                int bytesRead = stream.Read(buffer, offset, count);
                if (bytesRead == 0)
                {
                    throw new EndOfStreamException("Read beyond end of file EOF");
                }
                offset += bytesRead;
                count -= bytesRead;
            }
            return buffer;
        }
    }

Answer 3

I am not sure if this helps, but take a look at Increased Performance Using FILE_FLAG_SEQUENTIAL_SCAN.

SUMMARY

There is a flag for CreateFile() called FILE_FLAG_SEQUENTIAL_SCAN which will direct the Cache Manager to access the file sequentially.

Anyone reading potentially large files with sequential access can specify this flag for increased performance. This flag is useful if you are reading files that are "mostly" sequential, but you occasionally skip over small ranges of bytes.

Answer 4

If you dont mind using a tool, ESEUTIL worked great for me.

You can check out this blog entry comparing Buffered and NonBuffered IO functions and from where to get ESEUTIL.

copying some text from the technet blog:

So looking at the definition of buffered I/O above, we can see where the perceived performance problems lie - in the file system cache overhead. Unbuffered I/O (or a raw file copy) is preferred when attempting to copy a large file from one location to another when we do not intend to access the source file after the copy is complete. This will avoid the file system cache overhead and prevent the file system cache from being effectively flushed by the large file data. Many applications accomplish this by calling CreateFile() to create an empty destination file, then using the ReadFile() and WriteFile() functions to transfer the data. CreateFile() - The CreateFile function creates or opens a file, file stream, directory, physical disk, volume, console buffer, tape drive, communications resource, mailslot, or named pipe. The function returns a handle that can be used to access an object. ReadFile() - The ReadFile function reads data from a file, and starts at the position that the file pointer indicates. You can use this function for both synchronous and asynchronous operations. WriteFile() - The WriteFile function writes data to a file at the position specified by the file pointer. This function is designed for both synchronous and asynchronous operation. For copying files around the network that are very large, my copy utility of choice is ESEUTIL which is one of the database utilities provided with Exchange.

Answer 5

Eseutil is a correct answer, also since Win7 / 2008 R2, you can use the /j switch in Xcopy, which has the same effect.