I wouldn't go so far as to actually upgrade to MVC3, make it work, then upgrade to MVC4, and make it work, etc. But, essentially, yes, you'll just need to follow the upgrade path. Each release has notes that define things that changed, were added, or were removed. The newer version's release notes don't go into what happened in the last release, so you have to sort of compile the notes together on your own. You evaluate your app against each release's notes and determine what changes if any you need to make in your app. Then, go ahead and upgrade directly to MVC5 and make those necessary changes.
And, of course, make sure you have a backup of some sort. If you're using version control (and why wouldn't you be?), create a new branch specifically for the upgrade, and of course, only merge it back in once you know everything is working properly. This is where having comprehensive tests comes in super handy.