Why do you need X.org?
I'd try Qt first. It can run without X.org via frame buffer. See this link: http://qt-project.org/doc/qt-4.8/qt-embedded-linux.html
Another alternative were to use http://directfb.de/
To start an application you just need to create a startup script in /etc/init.d/