I solved with a damn simple proportion...
float x = (float) (10000 * 2 * EC_Main.viewer.getAspect() * diff.x / EC_Main.viewer.getWidth());
float y = (float) (10000 * 2 * diff.y / EC_Main.viewer.getHeight());
Vec3 worldOffset = invOrientation.mult(new Vec3(-x, y, 0.0f));
Taking into account my projection matrix
Mat4 orthographicMatrix = Jglm.orthographic(-10000.0f * (float) EC_Main.viewer.getAspect(), 10000.0f * (float) EC_Main.viewer.getAspect(),
-10000.0f, 10000.0f, -10000.0f, 10000.0f);