First of all, if you're trying to match some character in the range A-Z, you need to put it in square brackets. This
.replaceAll("A-Z|ss$", "SS")
will look for the three characters A-Z in the source, which isn't what you want. Second, I think you're confused about what | means. If you say this:
.replaceAll("[A-Z]|ss$", "SS")
it will replace any upper-case letter at the end of the word with SS, because | means look for this or that.
A third problem with your approach is that the second and third replaceAll's will look for any ss that was in the original string, even if it didn't come from a ß. This may or may not be what you want.
Here's what I'd do:
String replaceUml = str
.replaceAll("(?<=[A-Z])ß", "SS")
.replaceAll("ß", "ss");
This will first replace all ß by SS if the character before the ß is an upper-case letter; then if there are any ß's left over, they get replaced by ss. Actually, this won't work if the character before ß is an umlaut like Ä, so you probably should change this to
String replaceUml = str
.replaceAll("(?<=[A-ZÄÖÜ])ß", "SS")
.replaceAll("ß", "ss");
(There may be a better way to specify an "upper-case Unicode letter"; I'll look for it.)
EDIT:
String replaceUml = str
.replaceAll("(?<=\\p{Lu})ß", "SS")
.replaceAll("ß", "ss");
A problem is that it won't work if ß is the second character in the text, and the first letter of the word is upper-cased but the rest of the word isn't. In that case you probably want lower-case "ss".
String replaceUml = str
.replaceAll("(?<=\\b\\p{Lu})ß(?=\\P{Lu})", "ss")
.replaceAll("(?<=\\p{Lu})ß", "SS")
.replaceAll("ß", "ss");
Now the first one will replace ß by ss if it's preceded by an upper-case letter that is the first letter of the word but followed by a character that isn't an upper-case letter. \P{Lu}
with an upper-case P will match any character other than an upper-case letter (it's the negative of \p{Lu}
with a lower-case p). I also included \b to test for the first character of a word.