The general answer is that it depends on the engine you are using. Some engines require power-of-two textures for the reasons you stated in your question. And since an atlas is really nothing but a texture, it holds to this rule.
Again, depending on the engine, the various sprites in your atlas may need to also be power-of-two (this is the so-called "grid system" for spritesheets). But I have worked with engines that don't care, as long as the atlas itself is power-of-two.
Web-Based Games
For web-based games in particular, just from personal experience I have never had to worry about power-of-two. I just made my images whatever size they needed to be. This makes sense because:
- It's the web :)
- Padding images to powers of two increases file size and therefore bandwidth usage (though not by much)
I think (could be wrong though) most JavaScript engines don't care about power-of-two.