It's possible to set up a *ix to remote display to a system you're sitting in front of, without an X server on the machine running the app.
You need X11 libraries, one or more X11 applications, and you probably want openssh set up for remote X11 forwarding - all on the raspberry pi. You'll also want an X server running on the machine you're sitting in front of.
On the Raspberry Pi, change your sshd_config to include "X11Forwarding yes". After making this change, restart sshd or reboot.
Then apt-get or yum your X11 libraries and app(s).
When you connect to the Raspberry Pi machine, use "ssh -Y" instead of just ssh to pass along xauth data.
Good luck!