Well, I feel stupid. I've been working on this for a few hours now, and I missed a little step:
position += (k1.position + 2.f*(k2.position + k3.position) + k4.position)/6.f;
velocity += (k1.velocity + 2.f*(k2.velocity + k3.velocity) + k4.velocity)/6.f;
should be
position += (k1.position + 2.f*(k2.position + k3.position) + k4.position)*dt/6.f;
velocity += (k1.velocity + 2.f*(k2.velocity + k3.velocity) + k4.velocity)*dt/6.f;