Here's how I finally resolved it:
In addition to modifying the matrix, I had to normalize the color map into the 0 to 1 space. I also had to multiple by the length of the color map instead of 2^16.
Here's what that ended up looking like:
cmap = cmap - min(cmap(:));
cmap = cmap/max(cmap(:));
N = size(cmap, 1);
X = X - min(X(:));
X = (X/max(X(:)))*N;
X = uint16(X);
imwrite(X, cmap, 'test.tif');
I hope this saves some of you the trouble it caused me!