The use of prime numbers isn't about the difficulty of factoring - the most efficient method for factoring large composite numbers like the modulus of an RSA key is the General Number Field Sieve, which is much more efficient than simply trying to divide by successive numbers.
In fact, your statement that it doesn't work with composite numbers isn't really true - whilst RSA is generally performed using two primes, it still works more than two distinct primes, and since every composite number is a product of primes, you could choose two composite numbers (providing they are the product of distinct primes, and have no common factors) and the algorithm will still work.
For example, if we choose two composite numbers with distinct prime factors and no common factors:
a = 15 (= 3 * 5)
b = 77 (= 7 * 11)
This is equivalent to 4-prime RSA:
p = 3
q = 5
r = 7
s = 11
So far, so good however you go on to say that the "primes are your private key", which isn't really true either - the private key is actually the combination of the modulus n = p * q
(which is also part of the public key), and the private exponent, which is the number d
such that e * d = 1 mod phi(n)
where e
is the public exponent (which is often just 65537), and phi
is Euler's totient function. Once d
is calculated, the original primes can be discarded.
The problem is that calculating phi(n)
requires knowing the prime factors of n
. In ordinary RSA where n = p * q
we already know the prime factors of n
, but if we instead chose two composite numbers (like the a
and b
above) and use n = a * b
we need to factor these numbers in order to calculate phi(n)
. Since the security of RSA itself relies on the difficulty of factoring large composite numbers, it is infeasible to calculate the private exponent if we haven't constructed the modulus directly from its prime factors, which is the real reason that you have to use primes rather than composite numbers.