Compressing slow drive

[swiftie]

My largest drive is an external USB2 4Tb array. Both because of the USB2, and the fact that it's an array, it is my slowest storage, so it's used mostly as backup.

Now I have a fast quad-core system, I decided to compress this drive; I can afford the CPU for the decompression, and I'm reducing the poor I/O times in exchange for CPU which I have in abundance. In theory, the overall I/O should be faster.

Does this sound a good idea? Are there any pitfalls that I should be aware of?

The compression has been running for about four days so far, and is perhaps 33% complete. My system rarely reboots, but Windows Update might request a reboot.
What happens to the compression of the entire drive if it is interrupted part way through?

 

[AddRAM]

It`s never a good idea to compress a drive.

Why would you want to compress a storage drive anyway.

 

[DavidE]

USB2 is relatively slow compared to other current technology.
In your specs it shows a "Drobo 4-disk enclosure".
Does your Drobo device have any other connectivity options such as SATA, Firewire, USB3, ... ?
I found this thread about a "Drobo"
External HDD enclosure Drobo 4-disk SATA with USB and Firewire connection in Coventry | Hard Drives & External Drives for Sale | Gumtree.com

Just wondering if a better approach might be some type of hardware upgrade so you can use a faster access option...

I wouldn't want to add the complexity of compression - just one more thing that can go wrong...
4 days 33% done - seems like too much to me for what MIGHT be gained...

 

[swiftie]

Why would you want to compress a storage drive anyway.

Each block of data that is required has to be read from the drive. This involves seeking to the right cylinder, waiting for the block to go past the heads, then reading the data. All of these operations take aeons in "computer time". Then, in this case, the data has to travel via USB2 to get to the RAM in the PC. Another aeon.

By compressing the files, you will nearly always reduce the number of blocks involved by compression (that is, after all, almost its sole purpose). So compression will nearly always result in the data arriving in the PC faster. This time will be measured in milliseconds at the very least, often in seconds.

If the data is compressed, you then have to decompress it, which will take a few microseconds per block.

So, there are significant gains to be had in transfer speeds, if you can afford the CPU overhead.

When everything has settled down, I'll benchmark the I/O speeds, so I can quote actual numbers (this is complicated by caching mechanisms).

 

[swiftie]

Just wondering if a better approach might be some type of hardware upgrade so you can use a faster access option...

My Drobo is quite old (in IT terms). It has either USB2 or Firewire. The Firewire would be much faster, but all I read implies that it is unreliable at best, at least in the Windows environment. Perhaps this was because I was on XP at the time. I'd also need to install an adapter.

Compression was a free experiment. If I gain nothing but experience, it will still have been worthwhile.

All of my live data is on SSD, Internal SATA high performance drives, or USB3. But a free performance gain is not to be sneered at.

Back in the days of analog modems, people used to turn up their noses at compression, but it would often double your transfer rates. If you asked people if they wanted twice the speed at no cost, no one ever said no, and that was roughly the benefit you could expect on typical traffic.

 

[lehnerus2000]

Interesting

By compressing the files, you will nearly always reduce the number of blocks involved by compression (that is, after all, almost its sole purpose). So compression will nearly always result in the data arriving in the PC faster. This time will be measured in milliseconds at the very least, often in seconds.

If the data is compressed, you then have to decompress it, which will take a few microseconds per block.

So, there are significant gains to be had in transfer speeds, if you can afford the CPU overhead.

When everything has settled down, I'll benchmark the I/O speeds, so I can quote actual numbers (this is complicated by caching mechanisms).

It will be interesting to compare your "before" and "after" times to see if there is any benefit.

As a rule, I'm with AddRAM.
I try to avoid compressing files.

In fact, I've often wanted to have the ability to "spread data over a larger area", so that (theoretically) a normal storage bit sized error has less chance of corrupting my data (i.e. double the storage area used for a given bit of data).

Obviously a similar effect can be created by storing 2 copies of everything.

Back in the days of analog modems, people used to turn up their noses at compression, but it would often double your transfer rates. If you asked people if they wanted twice the speed at no cost, no one ever said no, and that was roughly the benefit you could expect on typical traffic.

You don't have to go back that far.

Allegedly Vista had terrible file copy performance when copying 1000s of small files.
Apparently if you created an archive (it didn't even have to be compressed) of those same files, the archive would copy much faster.

I never had a Vista install, so I can't confirm or deny this.