Cooja is indeed written in Java.
You can extend or modify Cooja if you need.
You can find out more about Cooja on the Contiki wiki as well as in numerous papres by Fredrik Österlind. Perhaps you should also take a look at Fredrik's PhD thesis "Improving Low-Power Wireless Protocols with Timing-Accurate Simulation", which is mostly about Cooja.