Use negative lookahead assertions:
/(X)(?!A{2})(?!B{2})(?:(A)|(B)){0,2}$/
Explanation:
(X) # Match and capture X
(?!A{2}) # Assert that it's impossible to match AA
(?!B{2}) # Same for BB
(?: # Non-capturing group:
(A) # Match and capture A
| # or
(B) # Match and capture B
){0,2} # Do this 0-2 times
$ # Then match the end of the string