Well, old devices may not be able to display text-only format properly as mentioned by samanthasquered. But, in my case mail-filters (spam filters or any filter created manually as mentioned by Mike W) would be a bigger concern.
Here is the best reference I could find about this topic: http://blog.hubspot.com/blog/tabid/6307/bid/32643/Why-Marketers-Must-Optimize-Emails-for-HTML-AND-Plain-Text-Infographic.aspx
Thanks,