According to the CSS features, HTML5 features, and misc. features that Modernizr detects, there is no mobile feature. I guess that would have to rely on a UserAgent string, which is not a smart idea. The whole point of Modernizr is to detect certain feature sets, which you can use to determine whether or not a feature of yours would work (for example, oh this browser doesn't have inline-svg so don't display this SVG; instead of, oh we think this is IE8 so let's not show this SVG).
I've come across multiple scenarios where my modals (pop ups) don't play nicely on small mobile / touch devices. In that case, I've always done something like this:
<a href="/register">Register</a>
<script>
$('a').click(function(e) {
if(!Modernizr.touch) {
e.preventDefault();
// show register modal
}
// fallback to page
});
</script>