Is hardware accelerated GUI data kept on the GPU

Question

I am doing some research as to how most hardware accelerated GUI libraries work. I am actually only care about the rendering backends of them here. I am trying to figure out what would be the best way to try and write my own as a sort of side project. I am trying to go for ultimate performance here instead of overly fancy features. I want to b able to draw primitives, text and animation.

Some good libraries that I know of are Qt, Skia and Cairo (though I'm not sure what the sate of HWA is on it). I have also looked at NanoVG which is a small library that seems to have a decent following. I did not manage to achieve decent performance with NanoVG though...

The one thing that struck me was that all these libraries seem to make use of the concept of "painting" where it seems that each primitive shape gets drawn from scratch over and over again. What I mean by that is that from the APIs, it does not appear as if though the shapes are created as "objects" on the GPU or whatever the terminology is and then left there to be rendered "automatically". In other words, they are not left on the GPU's memory for being redrawn in some large loop. To elaborate, it seems that for each i.e. rectangle that needs to be drawn, a whole OpenGL state is set up just to render that rectangle and is then destroyed again. It does appear though as if these rendered shapes are then at least rendered at their final destinations, allowing the GPU to then compose the entire scene.

The way I expected these libraries to work is by actually storing the entire scene on the GPU (excuse the horrible terminology). For instance, primitives would be triangulated and left in memory where after some intricate process would be used to have a main rendering loop for the scene. Furthermore there would then be mechanisms in place to update attributes or delete or add primitives. This is quite a vague description but I think that you get the idea.

What I would like to ask now, is if there is any performance benefit to the "painting" approach as compared to the "saved" approach (again, no idea if there are proper names for these things...). Some intricate caching mechanisms of sorts perhaps? Or is this just much simpler to work with?

I realize that the "saved" approach might use more memory on the GPU but are all the OpenGL calls needed for the "painting" approach not vastly expensive? I guess one might be able to compensate for this by caching the rendered shapes but does the GPU really provide one with so large a benefit when doing such a once-off (or not very regular) rasterization as compared to CPU, especially given the communication overhead? Also, does this communication overhead not pose serious problems for animations when drawing has to be done for every frame?

I am quite certain that NanoVG does not have an internal caching mechanism and I would assume that this could be responsible for its rather lackluster performance. Qt on the other hand seems to have excellent performance so it must be doing something right. Google also seems to be able to put Skia to good use.

PS. I am not a professional of any sorts and have only recently started to learn OpenGL.

EDIT: Another possibility that I have thought of is that maybe the "painting" approach was deemed necessary purely because of the memory benefits? The reason I would think that is because all these libraries were of course started in a different era and also target embedded platforms meaning that GPU memory might be so scarce on the target(ed) platforms and that using as little as possible of it might be more important than performance. Again though, in a situation like this, I am not convinced that frame-by-frame GPU rasterization given the communication overhead will outperform a CPU, especially considering the probably low pixel count on platforms with such little memory.

Furthermore I have read on http://blog.qt.io/blog/2010/01/06/qt-graphics-and-performance-opengl/ that Qt apparently blends together shader code from precoded segments at runtime before "painting" and then hopes for the OGL compiler to inline the code properly at runtime. This sound like even more OGL initialization overhead to me...

Answer 1

Saving the whole window as a single object into GPU (it would be bunch of rectangles saved as VBO) and then rendering it in a single OpenGL draw call would be fast, but it has several disadvantages:

• The whole geometry would have to be rendered using single shader. Having separate shaders (for opaque copy, transparent copy, gradient, ...) is more useful.
• The whole geometry could use only from limited amount of textures. Even if you use atlases, you need lot of textures for GUI. (Pieces of the GUI theme, icons, fonts, ...)
• You have to rebuild and reload the whole object into GPU after every change.
• Every widget must be able to produce their piece of geometry which is harder to abstract than 2D painting.
• On some GPUs you can render 2D stuff (filling area with color, copying from picture to picture, ...) with 2D commands which is faster than using 3D pipeline.

If you break it into several objects, then you eventually end up with single or few rectangles per object. It's easier and faster to render them without any stored objects.

What GUI frameworks do is tracking which exact parts of the window changed and repainting only them. The old image is cached in the GPU. This approach can be used with various drawing backends, not only OpenGL/DirectX accelerated rendering.

If you want to check example of a GUI library that does generate geometries which you can feed into OpengGL (or into different 3D api), look at librocket. It can actually bake static geometries together and render them in single draw call, but any element that will be changing often or need to render with its own shader has to stay separate.