You can use Device Administration API. And DeviceAdminSample is a good start.
There are following types you can set using setPasswordQuality()
DevicePolicyManager.PASSWORD_QUALITY_UNSPECIFIED,
DevicePolicyManager.PASSWORD_QUALITY_SOMETHING,
DevicePolicyManager.PASSWORD_QUALITY_NUMERIC,
DevicePolicyManager.PASSWORD_QUALITY_ALPHABETIC,
DevicePolicyManager.PASSWORD_QUALITY_ALPHANUMERIC,
DevicePolicyManager.PASSWORD_QUALITY_COMPLEX
There are several answers on SO