By doing so, I'm getting the individual commits from dev in the release branch
Probably because it is a fast-forward merge.
Simply do a:
git checkout release
git merge --no-ff yourDevBranch
And you will always have only one commit as a result of your merge (which can be problematic for git bisect though).