How long are you willing to wait to put this into production?
Right now the Linux graphics stack is built around Xorg. And Xorg has the inconvenient drawback, that you can't run purely off-screen X servers that make use of the GPU. If you can live with only one user making use of the GPU and the GPU holding the VT then you might want to look into Xpra
which you start with a X server configuration that uses the GPU instead of the dummy
driver.
If you're willing to wait another two years (hopefully) all drivers will fully support KMS and the DRM kernel interfaces; as much as I dislike certain aspects of Wayland, it's also a huge game changer that puts a lot of peer pressure on NVidia to finally get around and use the "standard" APIs. Already now you can use libgbm
to create purely off-screen OpenGL render contexts with GPUs that support it and no display server running; i.e. GPUs with open source drivers in the Mesa3D tree (Intel and AMD, however for now just OpenGL-3 and no OpenCL). Give it another 2 years and the APIs and tools will have stabilized that you can use this conveniently in production.