Your MDB defines the transaction border, the EJB B just takes part in the transaction started by A. A nested transation is something different.
All database locks are held until the transaction commits or rolls back, which is when A.onMessage() returns. So processWithLock will not release the lock after execution, when called within a global transaction.