As @AdrianMitev mentioned, you have to write java class to enforce new custom Password Policy.
- Create new class as mentioned in the doc.
- Build a jar
- copy it into
<IS_HOME>/repository/components/libs/
folder. - Then update the configuration file as mentioned in the doc.
HTH, DarRay