Ok, so I've done a whole lot more reading, And I'm just not sure that using a FileSystemWatcher is going to be viable. The size and time of these files coming in is just going to be too random and I'd rather not just have issues with events not firing etc.
I'm going to change my service to poll the directory(s) and get a list of files which I can then process sequentially without making excessive amounts of COM connections
Thanks to anyone that considered this problem for me