It sounds like you may have an issue with occasional "full GC" pauses.
I suggest you turn on GC logging, and then see if there is a correlation between these unwanted outages and the events in the GC log messages; read about the -Xloggc:file
option in the Java command manual entry.
Depending on what that tells you, the solution may be to adjust your JVM's GC options.