I think you'll eventually find that maintaining the paths relative to the js file will be harder, if even possible. When it comes time to ship, you are most likely going to want to concatenate all of your javascript files to one file, in which case you are going to want the templates to be relative to the baseUrl. Also, if you are fetching the templates via ajax, which Angular does by default unless you pre-package them in the $templateCache, you are definitely going to want them relative to the baseUrl, so the server knows where to find them once your js file has already been sent to the browser.
Perhaps the reason that you don't like having them relative to the baseUrl in development is because you aren't running a server locally? If that's the case, I would change that. It will make your life much easier, especially if you are going to work with routes. I would check out Grunt, it has a very simple server that you can run locally to mimic a production setup called Grunt Connect. You could also checkout a project like Yeoman, which provides a pre-packaged front end development environment using Grunt, so you don't have to spend a lot of time getting setup. The Angular Seed project is a good example of a Grunt setup for local development as well.