No, it is not possible as you've described.
Right now you have a processing server doing the CPU-intensive stuff and the workstation servers doing the GPU intensive rendering. They're tightly coupled over the X11 display connection. Both Xlib and OpenGL were designed on the assumption that the graphics card / screen weren't going to change underneath them, which is why you can't move to another workstation without restarting.
VNC attached to an existing X session works because it isn't doing any rendering at all. It's just reading pixels out of an existing window and copying them elsewhere.
With xfvb you are asking an OpenGL application to run on a not very good software renderer and it is refusing to do so. (Even if you could get it to work, the performance would suck.)
I would stick a decent graphics card and a monitor onto the processing server and run the OpenGL applications entirely on that server/display. Use x11vnc from the workstations to see the results.