You can simply delete them like you would any other file. For example:
rm MyHeader.h.orig
Alternatively, if there are no other untracked files, then after you commit with
git commit -a
you may clean your repository with
git clean -n
git clean -f
git clean -n
will tell you what git clean -f
will do, so you can be sure it's what you want.