One problem is that unsigned long long
is probably only 64 bits, and the intermediates result*base
and base*base
can easily overflow 64 bits in your modpow
function. You need an __int128
intermediate type here (if your compiler supports it).
Another problem is that RSA only works for plaintext/cipphertext values that map to integers that are NOT mulitples of p or q. If you choose a plaintext that is a multiple of p or q, the decryption will fail. This is not a problem with realistic (large) moduli, as the likelyhood of hitting a multiple of p or q randomly is astronomically small (its roughly the same as the chance of guessing the private key given only the public key).
The first problem is probably what is causing you grief.