How to reduce File IO on legacy app that writes to disk, and app that reads it in real time (.NET)

Question

My situation is I have a legacy app which I don't have the code for which writes out data to disk every second or so. I have a C# program I wrote which every second reads what was written to disk and uses the data. The data is written to a few text files which I know the file name before its created.

The issue is I have lots of virtual machines running this legacy app and my program. They are not limited by ram or cpu but I can't add more than 10 VMs per machine due to file io bottleneck.

Is there an easy way I can make a file on disk that exists in ram or something else? I heard something about named pipes being an option?

Thanks!

Answer 1

Are you sure of the actual IO involved?

Long ago I implemented a very ugly connection sending data from a dos program to a Windows program by means of a file. This was a lot faster than once a second, though--the dos program would send a 4k block anytime anything on it changed, 50 times a second (if it was caught up) the Windows program would read the frame number and then read the 4K block if the frame number differed.

This did NOT cause disk IO! You could sit there causing the dos program to update the frame many times a second for as long as you wanted and the hard drive light would stay off. Windows saw the file was open and being frequently written, the buffers were NOT flushed to disk until the updates stopped.

While I spent a lot of time optimizing the Windows side of the link it was all in what was done with the data, not in the connection--that simply wasn't a bottleneck despite it's apparent ugliness.

It's possible Windows would handle it differently if the file was closed each time. Sticking it on a ramdisk would keep it from doing the disk IO even then.

Answer 2

You can search for some kind of memory/temporary file systems.

• RAM drive for compiling - is there such a thing?
• http://256stuff.com/gray/docs/misc/linux_memory_tmp_filesystem_fs.shtml

I'm not sure if you can use pipes here since you legacy app is writing directly to the HDD.