SourceDataLine is pretty low level. It relies on native code implementations for each OS that Java accommodates.
The main way to cut latency is to make your buffer smaller.
There are other issues pertaining to Java audio and "real time" guarantees. You might find this article of interest: Real-Time Low Latency Audio Processing in Java.