If you are talking about formal language theory then of course all non-regular grammars (as in your example) can't be expressed with a regular expression (per definition).
But if you are wondering of what different regex flavors (in programming languages/regex libs) can do, then you can match all kinds of non-regular grammars/languages.
For example in Perl/PCRE you can match your example language with any of these:
Using recursion/sub pattern calls:
^(a(?1)b)$
Using a backreference (with a conditional):
^(?:a(?=a*(b(?(1)\1))))+\1$|^$
You may be interested in this questions and answers: Match a^n b^n c^n (e.g. "aaabbbccc") using regular expressions (PCRE)