Validation is considered failed only if you actually throw a javax.faces.validator.ValidatorException
Try throwing an Exception with the FacesMessage inside:
FacesMessage msg = new FacesMessage("my validaton message");
msg.setSeverity(FacesMessage.SEVERITY_ERROR);
throw new ValidatorException(msg);