It seems that this is the expected behavior.
To resolve this issue, instead of running multiple instances of the same activity within ParallelForEach, the workflow starts multiple instances of anther workflow. Therefore, ParallelForEach is no longer need, since individual workflow instances can run in parallel.