This configuration says how the transaction aspect will be applied. Briefly:
adviceMode=proxy, proxyTargetClass=true Cglib is used as proxy mechanism. If you use this, cglib must be on classpath, your proxied classes must have nonparametric constructor and they can't be final (cglib creates a child class as the proxy).
adviceMode=proxy, proxyTargetClass=false Jdk proxy mechanism is used. You only can proxy classes that implements a interface for methods that should be transactional. Jdk proxy can be type casted to the interfaces but can't be type casted as the original proxied class.
So, for adviceMode=proxy, the decision relies more on how are your code standards and what constraints result from used proxy mechanism.
adviceMode=aspectJ uses aspectJ library, which does byte code intrumentation instead of proxying.
adviceMode=aspectJ, compile-time weaving You should incorporate aspectJ instrumentation during a build process in your build scripts.
adviceMode=aspectJ, load-time weaving Instrumentation is performed on runtime. You have to put the aspectj agent as jvm parameter.
Using aspectJ is more powerful and probably more performant. It is also less invasive in terms of restrictions put on the classes you want to be transactional. However, proxy mode is simple Spring's out of the box solution.
More on proxies here http://docs.spring.io/spring/docs/3.0.0.M3/reference/html/ch08s06.html. More on aspectJ with spring here http://docs.spring.io/spring/docs/3.0.0.M4/reference/html/ch07s08.html.