I found out the answer. I was using the depth image, which isn't real world coordinates. I used the CoordinateMapper class in the Kinect SDK to transform the depth image into SkeletonPoints, which are real world coordinates.
It would go something like this:
using (DepthImageFrame depthFrame = e.OpenDepthImageFrame()) {
DepthImagePixel[] depth = new DepthImagePixel[depthFrame.PixelDataLength];
SkeletonPoint[] realPoints = new SkeletonPoint[depth.Length];
depthFrame.CopyDepthImagePixelDataTo(depth);
CoordinateMapper mapper = new CoordinateMapper(sensor);
mapper.MapDepthFrameToSkeletonFrame(DEPTH_FORMAT, depth, realPoints);
}