General pattern for failing over from one database to another using Entity Framework?

Question

We have an enterprise DB that is replicated through many sites throughout the world. We would like our app to attempt to connect to one of the local sites, and if that site is down we want it to fall back to the enterprise DB. We'd like this behavior on each of our DB operations.

We are using Entity Framework, C#, and SQL Server.

At first I hoped I could just specify a "Failover Partner" in the connection string, but that only works in a mirrored DB environment, which this is not. I also looked into writing a custom IDbExecutionStrategy. But these strategies only allow you to specify the pattern for retrying a failed DB operation. It does not allow you to change the operation in any way like directing it to a new connection.

So, do you know of any good pattern for dealing with this type of operation, other than duplicating retry logic around each of our many DB operations?

---

Update on 2014-05-14:

I'll elaborate in response to some of the suggestions already made.

I have many places where the code looks like this:

try
{
    using(var db = new MyDBContext(ConnectionString))
    {
        // Database operations here.
        // var myList = db.MyTable.Select(...), etc.
    }
}
catch(Exception ex)
{
    // Log exception here, perhaps rethrow.
}

It was suggested that I have a routine that first checks each of the connections strings and returns the first one that successfully connects. This is reasonable as far as it goes. But some of the errors I'm seeing are timeouts on the operations, where the connection works but the DB has issues that keep it from completing the operation.

What I'm looking for is a pattern I can use to encapsulate the unit of work and say, "Try this on the first database. If it fails for any reason, rollback and try it on the second DB. If that fails, try it on the third, etc. until the operation succeeds or you have no more DBs." I'm pretty sure I can roll my own (and I'll post the result if I do), but I was hoping there might be a known way to approach this.

Answer 1

Here is what I ended up implementing, in broad brush-strokes:

Define delegates called UnitOfWorkMethod that will execute a single Unit of Work on the Database, in a single transaction. It takes a connection string and one also returns a value:

delegate T UnitOfWorkMethod<out T>(string connectionString);
delegate void UnitOfWorkMethod(string connectionString);

Define a method called ExecuteUOW, that will take a unit of work and method try to execute it using the preferred connection string. If it fails, it tries to execute it with the next connection string:

protected T ExecuteUOW<T>(UnitOfWorkMethod<T> method)
{
    // GET THE LIST OF CONNECTION STRINGS
    IEnumerable<string> connectionStringList = ConnectionStringProvider.GetConnectionStringList();

    // WHILE THERE ARE STILL DATABASES TO TRY, AND WE HAVEN'T DEFINITIVELY SUCCEDED OR FAILED
    var uowState = UOWStateEnum.InProcess;
    IEnumerator<string> stringIterator = connectionStringList.GetEnumerator();
    T returnVal = default(T);
    Exception lastException = null;
    string connectionString = null;
    while ((uowState == UOWStateEnum.InProcess) && stringIterator.MoveNext())
    {
        try
        {
            // TRY TO EXECUTE THE UNIT OF WORK AGAINST THE DB.
            connectionString = stringIterator.Current;
            returnVal = method(connectionString);
            uowState = UOWStateEnum.Success;
        }
        catch (Exception ex)
        {
            lastException = ex;
            // IF IT FAILED BECAUSE OF A TRANSIENT EXCEPTION,
            if (TransientChecker.IsTransient(ex))
            {
                // LOG THE EXCEPTION AND TRY AGAINST ANOTHER DB.
                Log.TransientDBException(ex, connectionString);
            }
                // ELSE
            else
            {
                // CONSIDER THE UOW FAILED.
                uowState = UOWStateEnum.Failed;
            }
        }
    }

    // LOG THE FAILURE IF WE HAVE NOT SUCCEEDED.
    if (uowState != UOWStateEnum.Success)
    {
        Log.ExceptionDuringDataAccess(lastException);
        returnVal = default(T);
    }
    return returnVal;
}

Finally, for each operation we define our unit of work delegate method. Here an example

UnitOfWorkMethod uowMethod =
    (providerConnectionString =>
     {
         using (var db = new MyContext(providerConnectionString ))
         {
             // Do my DB commands here.  They will roll back if exception thrown.
         }
     });
ExecuteUOW(uowMethod);

When ExecuteUOW is called, it tries the delegate on each database until it either succeeds or fails on all of them.

I'm going to accept this answer since it fully addresses all of concerns raised in the original question. However, if anyone provides and answer that is more elegant, understandable, or corrects flaws in this one I'll happily accept it instead.

Thanks to all who have responded.

Answer 2

How about using some Dependency Injection system like autofac and registering there a factory for new context objects - it will execute logic that will try to connect first to local and in case of failure it will connect to enterprise db. Then it will return ready DbContext object. This factory will be provided to all objects that require it with Dependency Injection system - they will use it to create contexts and dispose of them when they are not needed any more.

Answer 3

" We would like our app to attempt to connect to one of the local sites, and if that site is down we want it to fall back to the enterprise DB. We'd like this behavior on each of our DB operations."

If your app is strictly read-only on the DB and data consistency is not absolutely vital to your app/users, then it's just a matter of trying to CONNECT until an operational site has been found. As M.Ali suggested in his remark.

Otherwise, I suggest you stop thinking along these lines immediately because you're just running 90 mph down a dead end street. As Viktor Zychla suggested in his remark.