Hardware Lossless Compression for Hard Drives? [closed]

Question

I happened across this article about hardware based hard drive encryption and realized that not only would this give a great way to protect your data but it would also speed up the applications that we use to encrypt that data.

This lead me to wonder...

Would it be possible to do the same thing for compression so that all of the data is compressed or uncompressed appropriately as it is read and written to the drive?

I haven't done any firmware programming in quite some time so I'm not even sure this is technically possible.

If it were, however, it could probably give quite a bit more storage space to folks.

What are the pros and cons of programming such an approach to be used in the firmware?

Answer 1

In the world of 1TB drives for $100 speed is a much more valuable resource than space. It wouldn't be worth it.

EDIT:

Ah so you're saying that it would be quicker to grab 100 bytes of compressed data off the platters, decompress it and then send it along to the system rather than grabbing 800 bytes of uncompressed data and send it along to the system because seek times are so slow.

That seems like a clever approach but I'm willing to bet that if the trade off ends up being worth it that hard drive manufacturers have already employed this technique and hard drive speeds are what they are in spite of the fact.

But who knows, you may be on to something!

Answer 2

I remember about 15 years ago seeing an advertisement for an IDE controller card that would do hardware compression. Not sure if it was any good or not. Those were the days when 1GB drives where over $1,000.

Answer 3

Who remembers Stacker? This was all already done-to-death in the '80's/90's. Speed was never a problem, and neither was it "tricky." It's just completely unnecessary these days.

Answer 4

As previously said, the gain is not that big, especially if you are storing seldom accessed files in a compressed form anyway.

As it would be hard to do in hardware (What disk size should be reported? What do you do if the entropy of the input equals its size?) and modern CPUs+RAM are blazingly fast compared to HDDs anyway, just do it in software.

An implementation I know of is compFUSed which is layered on top of any other file system, another one is ZFS Blog entry about how to enable it which supports compression natively.

Answer 5

I had also thought of this idea a while ago for network traffic- which has been done before: there are accelerator cards for comressing using gzip: http://www.aha.com/show_prod.php?id=36

I had also thought of another benefit would be that now you can transfer without compressing the contents from the drive- simply read from the disk the compressed blocks and send rather than having to compress at that later time.

Answer 6

It's possible, but it's very, very tricky. You'll have to develop custom drivers because while encrypted sectors are the same size as regular sectors, and thus use the same math to find data, compressed sectors are "smaller" so you either have to hold a map of 'real' sectors to compressed sectors in the OS, or on the drive itself.

The only other aspect is speed of access and latency. It shouldn't affect seek, but it may take longer to compress data than it would to write it - compression is fairly compute intensive.

Further, compression isn't really good until you get to large chunks of data. You can probably compress 512 bytes (1 sector) on the fly and get a few percent compression on average, but people really want to see 20% and more compression before they're willing to pony up the extra cash for the hardware.

It will require more on disk processing power and memory, which will increase the cost of the drive.

Further, drive capacity is growing at a rate that it's probably not cost effective to do this.

Let's say, for instance, that you develop the miracle compression that doubles the space with no performance drop, no extra (flaky or crash prone) drivers, works on any OS, etc. But it adds $100 to the cost of the drive.

It might make sense for someone to do this for a 1TB drive now, converting it into a 2TB drive, but in 6-8 months the 2TB drives will be below $200. It won't be worth it for any smaller drive, because you can get a 1TB for $99 now.

If you made it so it would work between the drive and the computer you're going to have much greater latency than building it directly into the drive, and the price/performance hit might not make it worthwhile.

So, technically it's possible, but has pitfalls and adds complexity and points of weakness to the system, but even if it didn't have these drawbacks, it is likely that it isn't worthwhile.

-Adam

Answer 7

One other consideration is that most of the big files on your disk (music, pictures and videos) are usually already compressed (MP3, JPEG, MP4/MOV), so compression wouldn't help those. And the files that aren't compressed (text files, word processing, text of emails) tend to be a drop in the bucket.

I was wondering the same thing myself because I was searching through thousands of gzipped text files, and ungzipping pegs my quad-core i7, and I wondered if dedicated gzip hardware could speed this up like a GPU speeds up parallel processing. But I'm suspecting the above concerns would make it so that for most uses a compressed hard drive wouldn't help much.