gl_Position.x and gl_Position.y is screen.x and screen.y coordinates of verticle i assume?
No, they're in clip coordinates. To get from clip to screen space you have to
- perform coordinate homogenization
p_NDC = p_clip/p.w
(now coordinates are in NDC space) - map NDC to viewport coordinates
p_window = (p_NDC + 1)/2 * viewport.{width, height} + viewport{x, y}
Another pitfall people often trip into is that OpenGL's matrix element ordering is column major, which "looks" (but isn't) transposed standard notation. I see you're using code from my linmath.h
library, in which I took care of that, i.e. calculations are done column major and if you fetch from and pass to OpenGL mat4x4
it's in the right order.