If you wrap your code in the following
using (TransactionScope sc = new TransactionScope(TransactionScopeOption.Suppress))
{
// code here
sc.Complete();
}
Then NHibernate will not enlist in the ambient transaction and therefore DTC will not have a dependency on the database transaction.
This is a hunch as you haven't supplied the error details in your question.
EDIT
Of course by following this advice your database commit will not be performed under the same transaction as the dequeue action so if there is a failure in your database this may or may not cause the dequeue transaction to roll the message being processed back onto the queue, so you risk dropping messages in this way. You can compensate for this in various ways or you can just run the risk if the cost of dropped messages are not high.