Short answer: no
It does not have a big impact on performance. The WP_Query object constructs MySQL queries for you and does not do any of the sorting itself, which means that performance decrease would be caused by MySQL and not Wordpress.
If the table(s) you're quering are extremly large, it might be worth it to look into MySQL database/table optimization using techniques such as table indexing.