Judging by your approach it seems that you are already found a way to get a list of Windows' processes from Java. In this case you can use one of the next solutions:
- Take other process info into account. E.g. command line or window title. Windows command line can be retrieved using
wmic.exe PROCESS
command and for Unix-bases systems byps -Af
. - When your program starts store it's process id in the registry or file in homedir. When another instance starts it should check that value and if process with that id alive then just exit.