When the OutOfMemoryError is accompanied by "Out of swap space?" the error occured while trying to allocate resources on a native level. In your case it seems an internal C++-procedure failed to allocate needed memory. This seems to be the reason why the heap-dump isn't created: when failing on the native level the VM's builtin error-handling might not have a chance to take over - as opposed to errors on java-level, like running out of (java-)heap-space.
By default, Windows NT user-space is 2GB, since your java heap is already 1,7GB there is not much headroom left for non-heap memory and your application seems to bump into the ceiling there.
There two things you can try to fix the error:
- reduce your Xmx to allow for more non-heap-memory
- try setting the /3GB boot parameter (not sure wether this is supported in your Windows-version) to reduce system-space to 1GB and so increase user-space to 3GB