Best way to track sql server 2008 database structure installed in client for next upgrade

Question

Our company has a product related to Human Resource Management developed in Asp.net and Sql Server 2008 & its installed in 20-30 clients. As the software is in continuous development due to each client's requirement.

Now when some new changes is being made & some client using older version ask for some changes then we have to installed the latest version code with latest db structure. Now we have to upgrade their database with the query & we have to track each client database structure for generating the query for upgrade.

Currently we are storing each client database & we generates sql to upgrade using the tool like Red Gate SQL Compare. As the client is being increased, this process is being very tedious so what are the alternatives for this process, can we use database versioning or any other alternative which automatically update the client db using the latest db reference using the installer like installaware?

Answer 1

I doubt you will find a single best answer to your question as there can be many ways to accomplish your goal, but I can share our process as well as a few insights I've gained over the past 10 years of doing what you've described. Perhaps they will help you identify a solution that works well for you.

First, updating a client database is a HARD problem. You never really know what they could have done to the database since you installed your software. I've had clients add their own tables, views and even indexes to our system tables because they thought the system needed them. All of this can create headaches when upgrading a target database because you don't necessarily know the state of the database to begin with.

Second, I like you initially thought that an installer to upgrade the database would be the way to go. Then I started thinking about the problem more and in our case at least the DBA often was not the one upgrading the front-end software. We also heard from a number of clients that installing anything on a server was a non-starter for them, so back to square one.

In the end, we opted to build our own executable that contained the scripts for the upgrade and the ability for the DBA \ end user to connect to the Server \ Database and run the upgrade process.

(In our case, we use lots of encrypted stored procedures which contain our proprietary business intelligence, which is why we don't just make the scripts available to clients.)

If you haven't already done so, get your database into source control! Visual Studio and Red-Gate each offer products to facilitate this. I've used both and don't particularly have a preference, but I'd say use what you and your team are comfortable with.

When we go to ship an update, we perform a diff from the latest version of the database in source control to an earlier version of the database. We look for any new objects and any changed objects - very rarely do we remove objects, but that might be something you need as well.

Once we know what's changed in the database we work very methodically through the list of changes to create idempotent scripts to perform the necessary changes. This is a very tedious process, but one that helps to ensure clients have an easy upgrade experience. Essentially what we do with this process is defensive programming - rather than assume a table, view, stored procedure, function, etc. exists in the database, we first check to make sure the item exists. If it does, we may be able to skip that DDL statement altogether. In other cases, we may want to drop the object and re-create it.

To be clear, we never drop a table, but we'll often drop other objects and re-create them to ensure the object is up to date.

The scripts get packaged up into our executable (a solution we built internally) in the correct sequence (e.g. if you are adding a table and a view that references the table you need the table to be created before the view).

As an aside, Red-Gate offers a tool that does this for you now, though I have not personally used it in production yet.

Once we're happy the update has been packaged properly, we try to break it! We test against all kinds of versions of our database - really old versions, databases that we've modified in one way or another, on all versions of SQL Server we support (e.g. 2000, 2005, 2008, 2012). If we find a situation that doesn't work we fix our scripts until it does.

I should also mention that our database schema has a log table which our update module writes to in order to log when an update was performed, who performed it, and which objects were part of that update. This has helped with troubleshooting on occasion.

While our utility doesn't make a backup of the database to upgrade prior to executing the scripts, we could, and I would recommend that you consider this as part of your process. (Our documentation stresses the importance of this and most of the DBAs we have worked with over the years do this naturally as well so we didn't feel the need to write this into the utility).

The database update utility gets packaged with our software updates and distributed as just another component of the overall product and can be executed by any user who has the proper credentials & permissions in our database at the client site.

While this process has worked well for us over the years, we continue to refine it as new tools are introduced in the market and as we encounter errors that clients experience. It can take a while to develop a process that works, so try not to bite off more than you can chew at any given time. For us, getting the database in source control was the biggest key. Once we did that things started to fall into place a little at a time.

Answer 2

Red Gate are looking at making deployments easier generally.

I'm a product manager at Red Gate and I've recently written a whitepaper called "Automated deployment using Red Gate tools" that presents one possible solution. http://assets.red-gate.com/delivery/deployment-manager/assets/pdf/automated-deployment-whitepaper.pdf

The approach is essentially to turn both web application and database components into packages that can then be deployed by a management system. The database is the hard part, and the paper suggests two approaches:

• Dynamic Upgrade. An upgrade script is created at the time of deployment. This is useful for development and testing environments where the database can easily be rebuilt. A database package contains the structure of the database, typically as scripts folder.

• Static Upgrade. An upgrade script is created when the package is created, typically at the end of a CI build, for each version that you know is currently deployed. (The management system keeps track of which versions are deployed.) At deployment time, the correct upgrade script is used. There are also pre and post deployment validation steps to make sure that the database hasn't changed, that the upgrade script will run fine and the upgrade script did run successfully. This method is really nice as it allows the upgrade script itself to be tested, as it is deployed through Development, Testing, Staging, Production, etc. It also allows the script to be reviewed by a DBA early in the process.

Answer 3

Consider supporting export/import functions.

The 'export' function essentially creates a backup of known/expected tables to a differently-named source. The 'import' function creates an empty copy of the latest version of objects and then imports rows from tables exported to the other source.

Whatever customizations they've applied, as Tim Lentine says they could do, they'd have to reapply. Your business will need to decide the extent to which they're willing to spend resource to reapply client customizations during an upgrade.

I'd tell the clients they need to make a backup first (assuming they have a DBA) or support a db backup function (if they don't) and then export and then import.

This assumes, of course, that your changes are always backwards-compatible.

It's a simple, cheap solution, if you can get away with it. If not, as Tim Lentine says, it can get really hard and messy.

HTH