The above described behaviour seemed very odd at first, but then i read about FlushMode and now it makes sense. The FlushMode of eclipselink as well as hibernate is set to FlushModeType.AUTO FlushModeType.AUTO automatically flushes Entities to the DB when it thinks it's neccessary. This can be because of an readable operation (Query) on a Persited (but not flushed) Entity but it also happened somehow randomly during my observations.
This breaks the rollback-on-failure mechanism, which I thought must be the standard behaviour of @Transactional.
To achive a propper rollback (on failure or if setRollbackOnly() is set), of all persisted but not flushed entities on transcaction commit, you have to explicitly set the FlushMode at the beginning of your Action.
JPA.em().setFlushMode(FlushModeType.COMMIT);
If you're using Eclipselink, you can also set the following property to make it default behaviour:
<property name="eclipselink.persistence-context.flush-mode" value="commit" />
Links which helped me understand: