In order to draw anything on X11 server display, one have to use X11 protocol. There is just no reason to do it in any other way then using xlib one way or another.
http://alleg.sourceforge.net/stabledocs/en/build/unix.html
Short version: whenever X available, xlib is used. If none present, it will try SVGA directly on console.