The simplest way is to get the Modernizr package that includes Yepnope.js. Then you can load conditionally this way:
Modernizr.load({
test: Modernizr.mq('only all'),
yep: '',
nope: 'plugins/respond.js'
});
Unfortunately, yes, you'll have to do this for every feature that you want to support with a polyfill. The yep
property allows you to load a script when the feature is present.