You are confusing several things here.
projMat * viewMat * modelMat * vPos
will transform vPos
into clip space, not screen space. However, using your coordinates as colors, you don't even want screen space (which are the pixel coordinates relative to the output window, and accessible via gl_FragmentPosition
in the fragment shader), but you want [-1,1] normalized device coordinates. You get to NDC by dividing by the w
component of your clipspace value.
From the images you get I can guess that projMat
is a projective transform, since in the orthogonal case, clip.w
will typically be 1 for all vertices (not necessarily, but very likely), and the results would look more like you would expect.
You can see from your image that x and y (r and g) are zero in the center. However, those values are not limited to the interval [-1,1], but your shader output values are clamped to [0,1] so you don't see much of a gradient. Your z coord is >= 1 anywhere, so it does not change at all in the image.
If you would use the NDC coords as colors, you would indeed see a red gradient in the right half, and a green gradient in the upper half, but the other areas where the value is below 0 will still be clamped to zero. You would also get some more info in the blue channel (although it might be possible that NDC z is <= 0 for your whole scene), but you should be aware of the nonlinear z distortions introducted by the divide.