So, it turns out that the reason this came up was because the merge was from the wrong branch. If you have two branches which are starting at the same spot:
A B
| |
| + Commit extra code
| |
This will happen:
git checkout B
git merge --no-commit --no-ff A
# Already up to date
git diff A
# Everything from the extra commit