File Compression

seekermeister said:

When I just formatted the 500GB drive, I selected the option to use file and folder compression. I wasn't certain if that was the thing to do, but I tossed a coin. Now, I'm wondering just how much it is able to compress a file...anyone know?

Yes, but I'm not sure you understand just how technical a question you just asked . NTFS compression uses something called a "compression unit" to determine the alignment and size of compressed byte ranges in a data stream (aka, a file or file segment). The default NTFS cluster size (if Windows formatted your drive) is 4K, so the CU size on a disk using those cluster sizes is 64K (16 clusters x 4K) - note that a volume with a cluster size larger than 4K cannot use NTFS compression (it can still do sparse files, but not NTFS compression).

Anyway, NTFS takes the file and compresses it - this is the actual compression - and then divides the file up into the compression unit. If the compression of the file results in the compression unit (again, default of 64K) being one or more clusters smaller than it was originally, that compression unit is written to disk compressed. If that section of the file stream does not compress smaller than 1 or more cluster, it's written to disk as-is, uncompressed. This makes for retrieval faster (parts can be uncompressed, and only the compressed parts need to be decompressed if needed) in random segments, but of course does make sequential transfer slower because you are going to have to decompress the compressed file stream units to copy the file.

Remember that NTFS compression is actually done at the file system allocation unit, but the type of data in a file still does matter - uncompressable data is still uncompressable data. If you have a movie file that doesn't compress very much when run through the high compression modes of WinRAR or 7zip, for example, the filesystem compression isn't going to get you much further than that either. NTFS compression is actually tuned for speed, not efficiency - keep that in mind when deciding if compressing your data at the filesystem would be any better than using a compression utility.

Another hint - movie files make *bad* compressed files - meaning, you're not likely to get much compression, if any, from a standard movie file. Either it's lossless and has little "empty" data regions, or it is a lossed container and is quite compressed in the file already, and as such is also not very "empty" either. Also given that a movie generally doesn't have the same byte pattern over and over, it doesn't compress well. You'd be better buying a larger drive as temporary storage, or see if the vendor will "cross-ship" you the new drive - you can wait until you receive the new drive (and copy data to it) before you have to send the old one back.

seekermeister said:

kado897 said:

The problem is whether they would compress enough to fit on the temporary drive without loss of quality. The normal archive formats such as zip and lzh are all lossless compression methods which would extract to the original quality. These types of archive hardly compress videos at all. The only way you can really make a video smaller is to reduce its quality. Once you have done that you can't get it back again.

I understand what you are saying, when it comes to the manner in which a video is originally wrapped. But I'm not talking about changing their original state, and not about using a zipper, but a compression method that is external to the files, such as I mentioned in formatting the drive, which is used by the OS, or a backup method used by something like True Image, which would also be external to the files. It is possible that neither of these methods would compress the files at all, if that is the case, then that is the answer to my question. If they would compress the files...as a group, rather than individually, would not the original quality be retained, after removing them from their compressed enviresonment?

I have to admit that I don't know how that compression works. I suspect the amount of compression would depend on how much disk space the NTFS file system wastes. I don't think it would be very much but you are correct in that you would not lose any quality.

Correct, but the compression ratio would likely be less than 1%. See my previous post.

cluberti said:

Correct, but the compression ratio would likely be less than 1%. See my previous post.

Our posts overlapped. That's about what I would have guessed.

I take it that these videos are not backed up then.

seekermeister said:

Not yet, that is what I'm trying to accomplish. Since I don't have a readily available HD large enough to back them up, the only other ways that I know of is to back them up to DVDs, but that would require ~150 disks for the job, or to use an online storage, but that would take an eternity to upload, and I don't know that any would be large enough for 750GBs of data.

EDIT: At the moment, I wish my Blu Ray player could write, but it doesn't. I guess a person could get some 50GB BD disks and write the videos on them. However, with the price of BD writers and media, I'm not going to do that.

Tricky but then I am paranoid about backups. I suppose that disks are more reliable than the used to be but...

Every Packer/Archiver allows you to split up bigger files into several small archives (which after extraction turn back into one large untainted file) But unlike on the pics you would want to set the compression to low/store when it's about precompressed material the few % of gain wouldn't justify the time used.

7-Zip

WinRAR


WinRar has even another helpful feature: it can add some sort of a checksum (recovery record) to the archives which helps you to repair the ones that might have gotten corrupted during file transfer

Alternatively you could try HJSplit which doesn't compress at all but splits every file into a number of smaller chunks



Edit Nah...forget this...750 GB it's not a size my recommendations are ideal for