How to trigger SPLIT's and DROP's when sharding in SQL Azure

Question

I am setting up a system running on Windows Azure for which I expect high volume of data and high traffic. In order to handle it, I am designing a Federated database. I am interested in having the application itself SPLIT (or DROP) federated databases when needed. There are 2 reasons that should trigger these operations to happen: 1) The size of the database is reaching the limit allowed in Windows Azure, and 2) The amount of traffic in the server is too high, and a SPLIT operation will improve performance, keeping the response time low (runs fast). (the inverse operations are based on similar reasoning).

My question is: How can I detect these 2 conditions programmatically?

Answer 1

You can use the Sql Azure Dynamic Management Views to programmatically monitor Sql Azure databases. Note that you will not be able to monitor the entire federated database at once, but rather each of its individual members.

Using the Dynamic Management Views to check for condition 1), the one related to size, should be straight forward. Detecting condition number 2), the one related to traffic / performance, is a bit more difficult since you will first need to identify the exact metrics that make sense and their threshold values.

One very important thing to keep in mind is that the SPLIT and DROP operations behave very differently. A SPLIT is an online operation (it does not involve any down time) through which a partition member is divided in two databases. The data is going to be automatically split between the two. This behavior means that splits might indeed be triggered from an automated scaling process.

The DROP however is quite different. When dropping a federation member, Sql Azure will move its range of key values to the lower or upper neighbor federation member, but the data itself is simply deleted. You can get a more detailed description in this article (search for "Scaling down" inside it). Basically you will have to manually export the data from the dropped database and manually merge it into the destination database. Technically speaking you might be able to automate the merge operation through the command line version of the Sql Azure Migration Wizard, but it's risky. It would require a lot of testing before putting it into production.

Microsoft is planning to implement automated merge on federation members drops, but that will happen in a future release. As it is at the moment, automated scaling down is not something I would recommend.

Update

For those interested, you can vote for the MERGE operation on federated SQL Azure databases here.