NServiceBus with NHibernate and MySQL

In the last project I have been working i finally got a chance to design and implement a solution based on NServiceBus and NHibernate, two tools I’ve been watching for a while but never got a chance to play with in more than sample applications. For some external reasons I’ve been forced to use MySQL as a database server in this project.

So basically I’m using NServiceBus to provide reliable communication between the involved components and NHibernate to do the persistence of the domain objects used by the components. Up to this point the hole design of the solution looks good, with minimal effort i have reliable, fault tolerant services that are ready to do their job.

Now i start implementing the details and get to the point where MySQL comes into play. I must say, it has surprised me… both ways.

The good thing is that after careful tuning, where my previous UNIX experience had a very important role, the database is able to handle the amounts of data that i plan to throw at it. Also it surprised me that some pretty complex queries run a lot faster than expected.

Then the bad things started to show up. The hardest to debug was that updating an index column from multiple parallel transactions causes deadlocks witch cause transactions to be aborted. Of course this only happens at high loads. It was not hard to avoid this after i found out what the problem was … but still after this i had a feeling of working with something that might not be as reliable as expected.

After that the MySQL .NET connector dropped the bomb on me: Distributed Transactions are not supported. Ok, they are not supported but why the hell does the connector throws an exception when used in a distributed transaction? I can understand that i can’t rely on the MySQL transaction being enlisted in the Distributed Transaction (DT) and that i have to handle that myself but not being able to use the connector AT ALL under a DT was unexpected. At this point i see only one solution: grab the source for the connector and modify the part that checks if a DT is present and just ignore it. Turns out this was very easy to do. If anyone is interested in this change in the connector i can provide more details.

Now i need to find a way of having a NHibernate ISession and a ITransaction per NServiceBus message handler.

The first approach was something similar to what Andreas Öhlund describes in this blog post. The only problem is that IMessageModule implementations in NServiceBus 2.0 are singletons and that was a problem because i need to store the ITransaction to commit or rollback at the end of the message handler. If in the next version of the NServiceBus there will be a way to have some message handler “wrapper” that it could work.

My solution was to use a base abstract class for the massage handlers. So instead of just implementing IMessageHandler<T> now i derive from this base class. The code below should speak for itself:

/// <summary>
/// Base class for message handlers.
/// Manages the unit of work required for handling the message.
/// </summary>
public abstract class MessageHandler<T> : IMessageHandler<T>
          where T : IMessage
{
    /// <summary>
    /// The injected unit of work implementation.
    /// </summary>
    private readonly IUnitOfWork unitOfWork;

    public MessageHandler(IUnitOfWork unitOfWork)
    {
        this.unitOfWork = unitOfWork;
    }

    /// <summary>
    /// Concrete handlers must implement this method.
    /// </summary>
    public abstract void HandleMessage(T message);

    public void Handle(T message)
    {
        try
        {
            HandleMessage(message);
            unitOfWork.Complete();
        }
        catch
        {
            unitOfWork.Abort();
            throw;
        }
        finally
        {
            unitOfWork.Dispose();
        }
    }
}

The implementation for the IUnitOfWork is in this case very simple, providing only the creation of the session and the transaction and the required operations. Since the unit of work is created per handler and the handlers don’t use other threads to do the work I don’t need to worry about making it thread safe.

public class MessageUnitOfWork: IUnitOfWork
{
    private readonly ISessionFactory factory;
    private readonly ITransaction transaction;
    private readonly ISession session;

    public MessageUnitOfWork(ISessionFactory factory)
    {
        this.factory = factory;
        session = factory.OpenSession();
        CurrentSessionContext.Bind(session);
        transaction = session.BeginTransaction();
    }

    public void Complete()
    {
        transaction.Commit();
    }

    public void Abort()
    {
        transaction.Rollback();
    }

    public void Dispose()
    {
        transaction.Dispose();
        CurrentSessionContext.Unbind(factory);
        session.Dispose();
        GC.SuppressFinalize(this);
    }
}

There is still one small problem. If the handler finishes it’s work without and exception and the mysql transaction is committed BUT an exception is thrown by the bus when committing the distributed transaction the MySQL transaction is not rolled back. But i realized that this only means that the same message might be sent again to the handler and that the handlers in general should handle this logical case since whoever sent the message is free to send it multiple times.

Since I’ve got this solution working it has handled a few millions of messages and there have been crashes and transaction that got rolled back occasionally but after all the system is designed to be fault tolerant and it has proven it is. Also in all the cases the database remained in a consistent state, witch in the beginning i was not sure it will.

In the end i would like to thank the NServiceBus team ( mainly Udi Dahan and Andreas Öhlund ) witch was very responsive and helpful on the support mailing list. I can only hope to find the time to contribute a few ideas to the next release of NSB. Also i would like to thank the NHibernate team for the great product they created ( can’t wait for the 3.0 ).

kick it on DotNetKicks.com


  • http://jonathan-oliver.blogspot.com Jonathan

    Although I haven’t verified this, one possible option would be to use the MySQL ODBC connector instead of the .NET connector. From what I understand it should support distributed transactions.

  • http://www.erata.net eti

    There is a bug on the MySQL connector’s bug-tracker in witch the developers claim that before distributed transactions are supported by the connector they have to be supported by the server. If this is true and the MySQL server and native library don’t support distributed transactions then i don’t think ODBC can support it. And for unknown reasons i don’t like ODBC :) .

    Anyway thanks for your suggestion.

  • http://jonathan-oliver.blogspot.com Jonathan

    From my preliminary tests the devart dotConnect driver for MySQL supports XA or distributed transactions. I’m still doing some tests to assert failure conditions but it’s working great. I’m also using the free version as well because I don’t require any of the other features when working through NHibernate or just pure SQL.

  • http://www.erata.net eti

    Jonathan,

    Thanks for the information. I would be great if the devart connector supported DTs. I think i’ll also play with it for a while and post the results…

  • http://www.erata.net eti

    From my preliminary tests it seems that indeed the devart connector supports, and works as expected when using distributed transactions. Disposing the distributed transaction without completing it does cause the executed statements to be rolled back even if they wore executed on multiple connections.

    The problem ( there always seems to be one ) is that the devart connector does not support starting a local transaction from the scope of a distributed transaction. Doing connection.BeginTransaction() directly or session.OpenTransaction() throws an invalid operation exception with the message “Local transaction can not be started while in a distributed transaction.”.

    I’ll try the devart forum to see if they have an answer for this…

  • http://jonathan-oliver.blogspot.com Jonathan

    One other possibility that we’re moving towards is to completely avoid a two-phase commit. Pat Helland who worked at Amazon wrote about the very same challenge that we’re talking about–processing a message from a queue and storing the results in a separate database. The results are found in his 10-page paper: Life Beyond Distributed Transactions: an Apostate’s Opinion. It’s a really good read and cuts right to the heart of the problem.

  • http://www.erata.net eti

    I’ve read Pat Helland’s paper, and it’s the ideas expressed are very interesting and made me try to see where and what the “entities” would be in the projects I’m working and how would i handle almost-infinite scalability.

    Still i think he discards as not scalable “large scale” distributed transactions and not “local” distributed transactions. By “local” DTs i mean that they happen on one machine or maybe on a cluster. If you need to write something to the db/storage and send a message (ex MSMQ) afterwords you still need to wrap the two operations inside a DT which uses a 2PC to do it’s job.

    Also I don’t think these “local” DTs affect the scalability of the entire system since they are … well “local” and part of the operations on a single “entity”.

    Anyway thanks for pointing the paper out, it was a good read. Here is the direct link to the paper.

  • http://www.watchfinder.co.uk Jonathan Adams

    Sorry to sound stupid, but where do I find IUnitOfWork ?

  • http://www.erata.net eti

    Hi, it’s just a simple unit of work interface:

    public interface IUnitOfWork: IDisposable
    {
    void Complete();
    void Abort();
    }

  • http://www.watchfinder.co.uk Jonathan Adams

    Yeah sorry I released that about 5 mins after I posted my message

    Could you point me in the direction of the change to MYSQL Connect please ?

    Thanks for the reply

  • http://www.erata.net eti

    1. Get the sources for the Mysql Connect.
    2. Open MySql.Data\Provider\Source\Connection.cs
    3. At line 286, right at the start of the EnlistTransaction add a return; statement or comment out the method body.
    4. Recompile

    This should do the trick.

  • http://www.watchfinder.co.uk Jonathan Adams

    Thanks !

  • http://www.watchfinder.co.uk Jonathan Adams

    That worked an absolute charm. I own many beers my friend ..

    Thanks !!

  • http://www.erata.net eti

    Glad to hear it helped.

    Anyhow, make sure you understand the implications of the change. The MySQL transaction is not enlisted in the distributed transaction and if something fails between committing the MySQL transaction and committing the distributed transaction, than the processed message will return to the queue to be re-processed but the MySQL transaction will remain committed.

    I’ve been using this in production on pretty heavy loaded systems and i’m yet to encounter a problem related to this.

  • Newbie

    Dear All,
    I have a C# application using the .net connector 6.2.3. So I am now inserting into a local db, one copy to a central db and one more copy to a copy of local db on the central server. How can I achieve this distributed transaction via .net connector? Any help or link will be a great help.

  • http://www.erata.net eti

    Hi,

    I have not done extensive testing but if i remember correctly the devart connector supports distributed transactions. You can download it here. ( the express version is free and should be enough )

    If you wrap all the db operations in a TransactionScope you should have all the operations enlisted in the distributed transaction.

    As a side note you could use NServiceBus with MSMQ to send/publish the data to be inserted to the remote machines. This way the distributed transaction would be local, coordinating the db call and the local msmq operation instead of remote.

    Good luck

  • Newbie

    Dear Eti,
    Is there any tutorial on how to use the devart connector? Sample codes? Thank you.

  • http://www.erata.net eti

    I believe all you need to do is change the referenced assembly to the devart one instead of the official mysql .net connector. Anyhow the docs are available here

  • Newbie

    Dear Eti,
    Where is the docs I cant see the link? where the link to download the devart connector? Can you show me please? So far is there any other people have use the devart connector? Are you the creator is it? Thank you.

  • http://www.erata.net eti

    In both cases the word “here” is a link.

    The download link is http://www.devart.com/dotconnect/mysql/download.html and the docs link is http://www.devart.com/dotconnect/mysql/docs/.

    At work, another team has used the devart connector the express version and i don’t remember them having any problems. Personally, I have only used it in a few tests so i can’t say much about it.

    On the version histoy page you can see that the first version was released in 2002 so i guess it’s “trustworthy”.

    And no, i’m not the creator of it :) .

  • Pingback: NServiceBus Distributed Transaction Woes