PHP has many functions that are either built in or available through extensions. However, there are some cases where libraries or applications were designed to call external programs. Without exec those packages simply aren't going to work, so you have your option of either allowing the use of exec or not using the package in question.
Forget about safe_mode: it's deprecated and about to be removed entirely.
The use of exec by itself is not inherently unsafe -- it's when programs don't properly sanitize input, or code has been written in a way that allows people to trick code into calling exec in a way it was not intended that represents the issue. As Drush is an administrative utility, you already have an expectation that this is something that will only be run by trusted users. I mean -- it includes letting you type in any php code you want and evals() it!!!