For smartphones, etc, you could move the media queries from with-in the css files to the link elements as shown below. This way you're only loading a single css file for each.
<link rel='stylesheet' media='screen and (max-width: 320px)' href='css/mobile.css' />
<link rel='stylesheet' media='screen and (min-width: 321px) and (max-width: 768px)' href='css/tablet.css' />
<link rel='stylesheet' media='screen and (min-width: 769px)' href='css/desktop.css' />
I don't think IE8 or less will support media queries like this so you should also provide a fallback:
<!--[if lte IE 8]>
<link rel='stylesheet' media='screen' href='css/desktop.css' />
<![endif]-->