Well, there are a few probabilistic algorithsm, some described in the wikipedia page, most likely you are speaking about Miller-Rabin Fermat Primality Test
Note that since 2002 there is actually a O(log(n)^6) deterministic approach to determine if a number is prime - called AKS (after its developers)1
It is an interesting issue - many thought that primality test cannot be done both polynomially in the size of the input (i.e. logarithmic in n
) and both deterministically, but their approach showed otherwise.