Why is mark-and-sweep GC not recommended in RMI system?
I don't believe it is.
Is it possible to run the GCs "Reference counting"-algorithm in a parallel thread without suspending the application itself?
While reference counting is not forbidden as a mode of GC, it is not supported by any JVM AFAIK as it has many limitations including performance, memory usage and circular references. I know C++ uses it but is a hack by comparison to what the managed memory systems do.
Note: MappedByteBuffers use reference counts for some purposes. This is an isolated use case.
There is a purely concurrent collector, the most popular of which is available from Azul. http://www.azulsystems.com/zing/pgc Note: it should really be called "pause less" instead of "pauseless" as it dramatically reduces GC related pauses, but doesn't eliminate them completely. (It is often used for low latency trading system in Java.)
If you are really concerned about GC pauses, the best thing to do is avoid using Java RMI. It is designed to be a "full fat" fully featured RPC which does lots of things you possibly never thought of doing. The Serialization isn't very efficient and generates lots of garbage. Using a more targeted RPC solution can reduce garbage by 90 - 99% or much better.