It is the purpose of the em unit that it is relative to the currently set font size. If you want to use an consistent form of em, use the unit 'rem'. It is relative to the root element of your page (most likely your html tag) and stands for root em.
Check out this article by Jonathan Snook if you want to learn more about it. http://snook.ca/archives/html_and_css/font-size-with-rem