Huge sql server database with varbinary entries

Question

We have to design an SQL Server 2008 R2 database storing many varbinary blobs.

Each blob will have around 40K and there will be around 700.000 additional entries a day. The maximum size of the database estimated is 25 TB (30 months). The blobs will never change. They will only be stored and retrieved. The blobs will be either deleted the same day they are added, or only during cleanup after 30 months. In between there will be no change.

Of course we will need table partitioning, but the general question is, what do we need to consider during implementation for a functioning backup (to tape) and restore strategy? Thanks for any recommendations!

Answer 1

Take a look at the "piecemeal backup and restore" - you will find it very useful for your scenario, which would benefit from different backup schedules for different filegroups/partitions. Here are a couple of articles to get you started: http://msdn.microsoft.com/en-us/library/ms177425(v=sql.120).aspx http://msdn.microsoft.com/en-us/library/dn387567(v=sql.120).aspx

Answer 2

I have had the pleasure in the past of working with several very large databases, the largest environment I have worked with being in the 5+ TB range. Going even larger than that, I am sure that you will encounter some unique challenges that I may not have faced.

What I can say for sure is that any backup strategy that you are going to implement is going to take awhile, so you should plan to have at least one day a week devoted to backups and maintenance where the database while available should not be expected to perform at the same levels.

Second, I have found the following MVP article to be extremely useful in planning backups which are taken through the native MSSQL backup operations. There are some large database specific options for the backup command which could assist in reducing your backup duration. While these increase throughput, you can expect performance impact. Specifically the options that had the greatest impact in my testing is buffercount, blocksize, and maxtransfersize.

http://henkvandervalk.com/how-to-increase-sql-database-full-backup-speed-using-compression-and-solid-state-disks

Additionally, assuming your data is stored on a SAN, you may wish as an alternative to investigate the use of SAN level tools in your backup strategy. Some SAN vendors provide software which integrates with SQL Server to perform SAN style snapshot backups while still integrating with the engine to handle things like marking backup dates and forwarding LSN values.

Based on your statement that the majority of the data will not change over time, inclusion of differential backups seems like a very useful option for you allowing you to reduce the number of transaction logs which would be have to be restored in a recovery scenario.

Please feel free to get in touch with me directly if you would like to discuss further.