You would need a another behavior for that state that said: (, (->( This transition should take you to another state(an auxillary state). That aux state's only transition would be ε, ε->(
From your original state you would need to pop ( off the stack when you see a ) ), (->ε
Here is a similiar example of mine that contains more a's than b's (problem number 1)