I've solved this now by using another way to load the key.
I've used this online converter to convert my PEM key to a XML key: https://superdry.apphb.com/tools/online-rsa-key-converter
So I changed the try { ... } part to:
rsa.FromXmlString("{XML_KEY}");
success = rsa.VerifyData(bytesToVerify, new SHA1CryptoServiceProvider(), signedBytes);
Now it works just fine. I guess I didn't really understand how to load a key via modulus and exponent.