Gravatar doesn't involve any significant server-side processing. The image URLs generated are just like any other, and might actually load faster than images on your own server (because they're from a different host and will load in parallel)
So no, it won't slow your site down in any unexpected way.
Edit: See this question regarding parallel requests on each hostname.