The problem is coming when you create str1 from the mutated bytes. Assuming your default encoding is UTF8, when you say String str1 = new String(b);
you're saying here are some bytes in UTF8 encoding, please build a nice string for me. But because you XOR'd the bytes, the encoding is invalid UTF8, and Java doesn't quite know what to do with it. If you look at the bytes that are being retrieved from str1 with b = str1.getBytes();
you'll see they are different than the bytes you created the string with!
Really you shouldn't be creating a string from "nonsense" bytes --- do you really need to store the XOR'd bytes back in a string?
If you really want to do that, you can trick the system by using a single-byte encoding where all the possible byte values are valid. Then you can be sure that the bytes you put into the string will be the same ones you get out. Here's an example that's working for me:
public class B {
static public void main(String[] args) throws Exception {
String str = "س";
System.out.println(str);
char key = 'N';
byte bKey = (byte) key;
byte[] b = str.getBytes("UTF8");
System.out.println("Original bytes from str:");
for (int i = 0; i < b.length; i++) {
System.out.println(b[i]);
}
System.out.println("Bytes used to create str1:");
for (int i = 0; i < b.length; i++) {
b[i] = Byte.valueOf((byte) (b[i] ^ bKey));
System.out.println(b[i]);
}
String str1 = new String(b, "Cp1256");
b = str1.getBytes("Cp1256");
System.out.println("Bytes retrieved from str1:");
for (int i = 0; i < b.length; i++) {
System.out.println(b[i]);
b[i] = (byte) (b[i] ^ bKey);
}
System.out.println("Bytes used to create str2:");
for (int i = 0; i < b.length; i++) {
System.out.println(b[i]);
}
String str2 = new String(b, "UTF8");
System.out.println(str2);
}
}
The output I get is:
س
Original bytes from str:
-61
-65
-30
-119
-91
Bytes used to create str1:
-115
-15
-84
-57
-21
Bytes retrieved from str1:
-115
-15
-84
-57
-21
Bytes used to create str2:
-61
-65
-30
-119
-91
س