Yes, you can use "arithmetic operator" in a HAVING clause.
The expression in the HAVING clause will be evaluated as a boolean.
NOTE: be sure the order of precedence is what you expect, e.g.
( COUNT(*) + 2 ) > 4
vs
COUNT(*) + ( 2 > 4 )
In your example, the addition isn't really required; the expression could be replace with the logically equivalent:
COUNT(*) > 2