See this article
"Setting a cancellation token allows you to abort Invoke (remember that when a delegate throws an exception, the exception is swallowed and only re-thrown by Invoke after all other delegates have been executed)."
Scenario 1. Imagine you have a user about to send messages to all ex-[girl|boy]friends. They click send and then they come to their senses and want to cancel it. By using the cancellation token they are able to stop further messages from going out. So if you have a long running process that is allowed to be cancelled, use the cancellation token.
Scenario 2 On the other hand, if you don't want a process to be interrupted, then use normal loop state exceptions so that the exceptions will be swallowed until all threads finish.
Scenario 3 If you have a process that is I/O intensive, then you probably want to be using async/await and not parallel.foreach. Check out Microsoft's task-based asynchronous pattern.