I don't think "nested" is the right term to describe this. These are three alternate transactions; none is nested within another. In particular, exactly one of the three is going to happen and be committed -- but which one happens is not deterministic. This one sentence should be enough to answer all three questions, but just to be sure, let's carefully say for each:
There's no guarantee. Maybe
takeTMVar b
will complete and commit; or maybe it will be pre-empted andtakeTMVar a
will be woken up and complete. But they won't both complete, that's for sure.Yes, that's correct: all three
TMVar
s can wake this thread up.The semantics don't change: whenever several of them can commit, the left-most one will. (In particular, the paper describing STM says, "The
orElse
function obeys useful laws: it is associative and has unitretry
.".)(from your question in the comments) The semantics of STM on page 8 of the linked paper really does guarantee that the left-most successful transaction is the one that succeeds. So: if thread
A
is executingtakeTMVar b
(but has not yet committed) and threadB
executes and commits a write toa
, and nothing else happens afterwards, you can be sure that threadA
will be restarted and return the newly written value froma
. The "nothing else happens afterwards" part is important: the semantics makes a promise about what happens, but not about how the implementation achieves it; so if, say, another thread took froma
immediately (so that thetakeTMvar a
is still going toretry
), a sufficiently clever implementation is allowed to notice this and not restart threadA
from the beginning of the transaction.