Re: Is there a simple way to check? The short answer is no.
To know what Devise gems to use, just follow the directions in the devise wiki - you just need devise and devise-encryptable according to the wiki.
Re: Which one to do first: Whatever you decide to do, its a good idea to have a stable branch before attempting an upgrade. So either you a) upgrade Rails + get restful_auth working, or b) use Devise + upgrade Rails - just avoid trying to do both at the same time.
a) seems possible if you use https://github.com/Satish/restful-authentication . It's a fork of the original project, and neither are as active as Devise ( https://www.ruby-toolbox.com/categories/rails_authentication ). You may choose to upgrade your authentication system to devise anyway, which leads us to...
b) probably the road-more-travelled. You should be able to find plenty of (Rails 2.3) upgrade instructions for restful_authentication->Devise. Get your specs green, then attempt the Rails 3 upgrade.