"Stop the existing instance" : The Task Scheduler service will stop the instance of the task that is already running, and run the new instance of the task.
that means it will kill the process that the scheduler has launched, cmd.exe
in your case, as you told us your program is started from within a batch.
Now, i'm not familiar with java but i guess that stopping your batch will kill the java process that where launched if it is not started as a service.
The scheduler will then run another cmd process and execute your batch once again