You're probably waiting for the connections on the other tasks to fail.
Try setting the connection timeout to say 2 seconds.
Stop does not forcefully stop the other tasks, it just prevents new tasks from starting. Parallel.ForEach decides how to partition the work among tasks.
A better option would be to use Connection.OpenAsync
with a cancellation token and use Task.WaitAny()
.