I had similar issues in a Interop-PowerPoint scenario. I never found the exact reason, but a likely supsect is the number of open COM-Objects. Are you releasing all Objects properly?
Have a look at this thread for details: How do I properly clean up Excel interop objects?
As a last resort you could implement a watcher thread which restarts your process. Not nice, but it may "solve" the problem.