If you wrap your code in the following
using (TransactionScope sc = new TransactionScope(TransactionScopeOption.Suppress)) {
NHibernate will then not participate in an external transaction, and therefore DTC will not be dependent on the database transaction.
This is a hunch since you did not provide details of the error in your question.
EDIT
Of course, following these tips, your database commit will not be performed within the same transaction as the dequeue action, so if your database crashes, it may or may not cause the dequeue transaction to roll the processed message to queue, so you run the risk of discarding messages this way. You can compensate for this in various ways, or you can just take a chance if the cost of the discarded messages is low.
source share