Basically, you are doing all right, but:
str.replace(target, replacement);
must be replaced with
str = str.replace(target, replacement);
because replace doesn't change the string itself, but returns a "replaced string".
P.S.: German has more special characters than "ö"; you're missing "ä", "ü" (and their corresponding capital letters), "ß" etc.
And here's my test code:
package test;
public class Test {
public static void main(String[] args) throws Exception {
String latexText = "J{\\\"o}rg";
String normalText = replaceBibtexMutatedVowels(latexText);
System.out.println(latexText);
System.out.println(normalText);
}
public static String replaceBibtexMutatedVowels(String str) {
CharSequence target = "{\\\"o}";
CharSequence replacement = "ö";
str = str.replace(target, replacement);
return str;
}
}