You can check for error like this
if (#{facesContext.maximumSeverity == null}) { openConfirmDialog(event.data); }
In case if you have more than 1 type of messages (error, warning, info) you can use
if (#{empty facesContext.messageList}) { openConfirmDialog(event.data); }
for checking validation messages before showing the dialog.
Example of usage in oncomplete
:
oncomplete="if (#{facesContext.maximumSeverity == null}) {#{rich:component('popup')}.show();}"
In case you want to react on warnings and errors but not on info messages use
oncomplete="if(#{facesContext.maximumSeverity.ordinal gt 0}) {#{rich:component('popup')}.show();}"
Additional information about last case here.