If on Windows I'd suggest simple plain VNC on desktop machine. Yes I know it sounds stupid. For a terminal server kind of access NVidia provides apropriate combinations of hard and software.
On Linux there's another possibility: Xpra. Simply spoken Xpra is a special kind of compositing manager, that uses a different display as the composition surface. It can operate over low bandwidth links by using efficient video codecs for compression. Usually Xpra uses a virtual framebuffer X server to run the clients on. But is can also be used in combination with a X server using a GPU.
However each user uses it's own X server for this. So if the system's GPU is used only one user can use Xpra at a time (actually since the NVidia drivers claim supporting hybrid graphics now, it may be possible to exploit that somehow – I haven't researched into that yet, though).
So the user starts Xpra using
xpra --start-child=xterm --vfb=/usr/bin/X start :100
And can then connect to the remote machine
xpra attach ssh:remote-host:100