SQL Server 2008, Compression / Zip and fast query

Question

I have an application that produce approximately 15000 rows int a table named ExampleLog for each Task. The task has a taskID, that is saved in a table named TaskTable, thus it's possible to retrieve data from the ExampleLog table to run some queries.

The problem is that the ExampleLog table is getting very big, since I run everyday at least 1 task. At the time being my ExampleLog table is over 60 GB.

I would like to compress the 15000 rows which belong to a TaskID, and compress them or just Zip them and then save the compressed data somewhere inside the database as Blob or as Filestream. But it is important for me to be able to query easily the compressed or zipped file and proccess some query in a efficient manner inside the compressed or zipped data. (I don't know, if it's possible or I may lost in term of performance)

PS: The compressed data should not be considered as backup data.

Did someone can recommend an good approach or technique to resolve this problem. My focus is on the speed and of the query running on the ExampleLog and the place taken on the disk.

I'm using SQL Server 2008 on Windows 7

Answer 1

The answer of Denis could not solve my Problem completely, however I will use it for some optimization inside the DB. Regarding the problem of storing data in package/group, there are 2 solutions of my problem:

• The first solution is the use of the Partitioned Table and Index Concepts.

For example, if a current month of data is primarily used for INSERT, UPDATE, DELETE, and MERGE operations while previous months are used primarily for SELECT queries, managing this table may be easier if it is partitioned by month. This benefit can be especially true if regular maintenance operations on the table only have to target a subset of the data. If the table is not partitioned, these operations can consume lots of resources on an entire data set. With partitioning, maintenance operations, such as index rebuilds and defragmentations, can be performed on a single month of write-only data, for example, while the read-only data is still available for online access.

• The second solution it to insert from the code (C# in my case) a List or Dictionary of row from a Task, then save them inside a FILESTREAM (SQL Server) on the DB server. Data will later by retrived by Id; the zip will be decompressed and data will be ready to use.

We have decided to use the second solution.

Answer 2

Consider Read-Only Filegroups and Compression.

Using NTFS Compression with Read-Only User-defined Filegroups and Read-Only Databases

SQL Server supports NTFS compression of read-only user-defined filegroups and read-only databases. You should consider compressing read-only data in the following situations: You have a large volume of static or historical data that must be available for limited read-only access. You have limited disk space.

Also, you can try and estimate the gains from page compression applied to the log table using Data Compression Wizard.