Your example only looks pathological, because you are using an overly specific example.
In practice, you don't "return" >> identifier;
. Usually, the thing that's returned is just an expression. So, you'd say
expr = literal | variable | function_call;
Now the general way to cater for parenthesized expressions in on fell swoop is simply:
expr = literal | variable | function_call
| ('(' >> expr >> ')')
;
Bam. Done. It handles the balancing. It handles nested parentheses. It handles (((foo)))
even. Not a whistle was given that day.
I don't think there is /anything/ wrong at all. I've posted probably over 20 recursive different expression grammars in answers on this site. They should provide motivating examples (showing operator precedence and overruling them with these parentheses).