As you are using strings representing hexadecimal numbers you should tell that to Math_BigInteger
$rsa = new Crypt_RSA();
$rsa->loadKey(
array(
'e' => new Math_BigInteger($exponent, 16),
'n' => new Math_BigInteger($modulus, 16)
)
);