You need more memory most likely. Dotfuscator isn't exactly lightweight in memory when running through large projects. I'd recommend 2 gigs as a minimum for most projects, and 4 gigs preferable. Also, it might run faster under 64-bit, due to the extra virtual memory space. Also, you might want to ensure that you're losing the latest version of Dotfuscator, since there have been some performance improvements between releases in the past
Excludes usually won't speed up Dotfuscator significantly unless you're excluding whole assemblies.