I raised the issue against hammer.js on github in case it was a bug and got the answer I was looking for: https://github.com/EightMedia/hammer.js/issues/532
It turns out that hammer.js has a default option 'stop_browser_behavior.userSelect' which is set to 'none'. It's not clear why it only works for IE but it does state in the comments related to the default 'userSelect' value:
// this also triggers onselectstart=false for IE
and my guess is that this has something to-do with it.
Adding the following javascript at the start of my code resolves the issue:
Hammer.defaults.stop_browser_behavior.userSelect = '';