You need a running X server, so that X11 based programs can display output. However if your goal is OpenGL development on Windows then you should not use a X server, because, unfortunately, none of the existing X servers for Windows does support modern OpenGL features in their implementation of GLX client and server components.
I strongly suggest you target native Windows GDI when compiling for windows.