If the gem does not provide any helpers or generators, then it's typically just a convenience that allows you to easily update the version of the assets via the Gemfile. This is nice, as long as the gem itself is relatively close in version to the actual source.
I usually start with the gem, make sure it's being updated regularly, and then move to the source if I really need to.
I've also recently started using bower for some JS sources, and there is a complementary bower-rails gem that provides some bundler like capabilities. Pretty sweet actually!