You probably need to at least invoke 'xhost +' from a terminal invoked in the graphical shell. There's probably some way to make that the default, but generally X11 disables access from other consoles that aren't the GUI login console until you invoke that command.
To get it to work at boot time, you can check out this thread or this thread. I think what would likely work best for you is to add to your .xinitrc file:
xhost +local: