This example appears relevant to your question: Tansactions in WCF
It has a client calling into 2 WCF services in a transaction. Each service itself issues a transaction on a DB. The example tests the case where the call to first server suceeds but then an exception is thrown - no data from first call is written to db as client transaction is rolled back due to exception after first server call.
The question of how this works is a good one as there does seem to be an element of magic involved! I'm no expert on this (so probably better to wait to hear from one!) and all I can suggest is that when you declare your WCF method [OperationBehavior(TransactionScopeRequired = true)]
then WCF does some reflection magic on the method code to wrap database activity in a transaction - and it handles rolling that generated transaction back on the db on exception - but I can't give insight into the details of that mechanism (I'm sure someone else can) - you might want to reword a new question that focuses more clearly/directly on that point.
You could also gain insite by experimenting with the functionality of your service method - i.e if you change the db code to instead write a file you will probably find that the file is written if there is an exception or not.
Update: There is some detail about how TransactionScope works in this answer.