The documentation is not very clear on this point, but my interpretation is as follows. This is based on Glassfish v4 reference manual.
Password encryption algorithm determines how the passwords are encrypted within your database. This is the parameter digestrealm-password-enc-algorithm
. You really want to have this set to something because of course leaving passwords in a database in the clear is a security hole.
When someone tries to authenticate, glassfish needs a way to compare what was submitted to what's in the database. But, since the latter is all locked up, it needs a key to unlock. The encryption (strictly, hashing) used on that key is what is defined in Digest Algorithm (parameter digest-algorithm
). It defaults to SHA-256 in v4 (prior, it was MD5).