=COUNTIFS may be best (if available) but with =COUNTIF I'd suggest:
=COUNTA(A:A)-COUNTIF(A:A,"Yes")-COUNTIF(A:A,"No")-COUNTIF(A:A,"=""").
If you are counting yes
and no
anyway then you could use the results from elsewhere in place of parts of the above formula.