The SERIALIZABLE transaction isolation certainly is the most certain way to achieve your goal but it could mean that performance will suffer.
There is one option you have not considered and that is to build your own semaphore.
You could create a static ConcurrentHashMap of items currently being processed and (at the start of each insert process - put a record and when done delete it.
Then each Thread process could consult this semaphore before starting inserts.