You can set the styleClass
conditionally ,like this
<ace:textAreaEntry
styleClass="#{facesContext.validationFailed?'failedClass':'validClass'}"/>
or assign fail class only and otherwise no class at all
<ace:textAreaEntry
styleClass="#{facesContext.validationFailed?'failedClass':''}"/>
If you manually set message to be displayed in your page you can check if facesContext.messageList
is empty or not, like this
<ace:textAreaEntry
styleClass="#{(not empty facesContext.messageList)?'filedClass':'validClass'}"/>
If you want to test for some specific internal logic validation you can check for some boolean for example
<ace:textAreaEntry
styleClass="#{(myBean.someComponentFailed)?'filedClass':'validClass'}"/>
where someComponentFailed
is some property that you set to true/false upon validation failure