GLSL Orthographic projection matrix not working like it's supposed to

Question

When I try to use my orthographic projection, I'm not getting the result I'm looking for. I have a VBO containing the following 2D vertices and texcoords (every other line):

0, 0,
0.0, 0.0,
512, 0,
1.0, 0.0,
512, 512,
1.0, 1.0,
0, 512,
0.0, 1.0

At the moment, I draw them using glDrawArrays(GL_TRIANGLE_FAN, 0, 4); There doesn't seem to be any problem with the VBO (I've checked it in gDEBugger), instead the problem seems to lie in the vertex shader.

This is the offending shader:

#version 420

uniform mat4 projection;
uniform mat4 modelview;

layout(location = 0) in vec2 vertex;
layout(location = 1) in vec2 texcoord;

out vec2 f_texcoord;

void main() {
    const float right = 800.0;
    const float bottom = 600.0;
    const float left = 0.0;
    const float top = 0.0;
    const float far = 1.0;
    const float near = -1.0;

    mat4 testmat = mat4(
        vec4(2.0 / (right - left), 0, 0, -(right + left) / (right - left)),
                    vec4(0, 2.0 / (top - bottom), 0, -(top + bottom) / (top - bottom)),
                    vec4(0, 0, -2.0 / (far - near), -(far + near) / (far - near)),
                    vec4(0, 0, 0, 1)
    );
    gl_Position = testmat * vec4(vertex, 0.0, 1.0);
    f_texcoord = texcoord;
}

I'm not overly familiar with transformation matrices so I'm learning right now, and from what I have read the matrix above is correct if I want orthographic projection, which makes me very confused as I can't get it to work.

Here is an image to illustrate the problem: (Note that the texture has transparent parts.)

EDIT: This is how it looks with the modified values:

Answer 1

Matrices in GLSL are column major. The first vec4 should be the first column, not the first row. The last coordinate should be zero.

mat4 testmat = mat4(
    vec4(2.0 / (right - left), 0, 0, 0,
    vec4(0, 2.0 / (top - bottom), 0, 0),
    vec4(0, 0, -2.0 / (far - near), 0),
    vec4(-(right + left) / (right - left), -(top + bottom) / (top - bottom), -(far + near) / (far - near), 1)
);

This uses the standard OpenGL glOrtho transform. The origin will be at the bottom-left, with +X going right and +Y going up. That's standard for OpenGL.

If you want a top-left origin, with +Y going down, you'll have to adjust the matrix accordingly.

My code for doing so is as follows:

Translate(-1.0f, 1.0f, (depthRange.x + depthRange.y) / 2.0f);
Scale(2.0f / size.x, -2.0f / size.y, 1.0f);

Where depthRange.x/y are the zNear/zFar values. size.x/y is the width/height of the window. Translate creates a translation matrix, and Scale creates a scale matrix (they are right-multiplied with each other).

Answer 2

the problem is - multiplying matrix. change:

testmat * vec4(vertex, 0.0, 1.0);

to:

vec4(vertex, 0.0, 1.0) * testmat;