Thanks for the tip Stuart. This is indeed not a MvvmCross issue but rather a general Android issue.
The solution was to add the following to each clickable item:
android:focusable="false"
android:focusableInTouchMode="false"
This comment got me on the right track https://stackoverflow.com/a/15859831/83301.