It's not a question of "why is the server swapping" so much as "why is my app eating so much RAM?"
STRONG SUGGESTION:
Download the Microsoft CLR profiler:
To your original question "Why does the server use swap even if it should have plenty of free RAM?" - there are many possible reasons. You'll need to have administrative access to the server to determine exactly what's going on. These links might help:
https://serverfault.com/questions/305295/why-does-windows-2008-use-swap-before-the-memory-is-full
http://blogs.technet.com/b/askperf/archive/2008/03/25/lock-pages-in-memory-do-you-really-need-it.aspx <= You need to consider not just your app, but other things running on the server as well...
- *