What people don't seem to grasp is how well these things can be tampered with by someone who would be considered a top notch programmer in the industry. They know how to falsify time stamps as well as other things giving the appearance of the genuine article by simply changing some of the coding itself not the size of any particular file.
That's precisely the reason that even better "programmers" (these folks like to be called computer scientists, thank you) came up with the concept of a
digest. Now read carefully because hopefully I'm going to put this one to bed once and for all. It does require some understanding of mathematics so get your thinking caps on!
When virus "research" was in it's infancy many methods were found for adding a virus payload to a file without
looking like you'd added a virus payload to a file. One way was to use a resource editor to carefully remove some non-code content like icons, etc. from the end of a PE (portable executable) image and then append your virus code to the file after carefully padding it out to the exact same file size. As filesystems became more robust folks were even toying with putting malware in NTFS alternate data streams. Needless to say so many ways to tamper with a file while leaving the same filesize were found as to make filesize alone a poor indicator of file integrity. Some other way was needed -- enter the checksum...
When serial communication was in it's infancy hardware limitations and just plain poor electrical design combined with legacy limitations of connected devices like modems, etc. brought the term "data corruption" into the vernacular of every computer geek at the time. When you sent data across a wire from one device to another there was a chance that what you would receive sent was subtly different than what you sent. That's hardly ever what you want
And what's more, the problem seemed to increase as the square of the distance so a cable run twice as long was four times as likely to have transmission errors. Those "computer scientists" (not programmers, PLEASE!) also dabbled in mathematics. The main two parts of mathematics that are applicable to this problem are number theory and probability.
The obvious way to check for data corruption is to compare what you got with what you sent byte-by-byte. Sounds like a really simple easy solution, problem solved, let's move on. Not so fast... To do this comparison assumes that you have a copy of the original dataset on both sides of the serial link. So think about it, if you had that, you'd not need the serial link so much now would you? One of those bright beanie-wearing computer scientists turned to simple math for an inexpensive method for determining integrity of data moving across a serial link.
Parity was thus born as is defined as the use of XOR (eXclusive OR -- a boolean operation) across fixed groups, or words, of data to create a parity word that can be used to detect errors in transmission by reapplication of the algorithm at the receiving end. I'll dispense with the proof here and let it suffice to say that single bits of change are detected flawlessly. Two-bit changes are detected at a rate of 1/n. Combine those rules and changes to odd number of bits are always detected and even numbers at a remarkably efficient rate but still not a guarantee of integrity. We need more math...
Another algorithm know as a modular sum was introduced and widely used in early maninframes whose error detection rate was somewhat better than 1/n. It was used for a while until fixed size transmissions fell out of fashion and data transposition, null drop and truncation errors all came into play. A notable figure at this stage is JPLs Mark Adler, author of the zlib compression library and inventor of the Adler-32 checksum algorithm. His modification of the older Fletcher algorithm was a true step forward in computer science of the time due to it's speed and improved ability to catch errors. But it wasn't enough. Enter CRC-32.
This could go on and on but I think you get the point. More and better error correction algorithms have come over time. CRC-32 is even a part of the TCP suite of protocols that make the internet run. With the advent of cryptography more math is brought to bear resulting in things like a SHA-1 sum and other methods principally developed by computer scientists at RSA Data Security. As those algorithms have improved the concept of "statistical uniqueness" has arisen.
Two datasets that are too large to compare byte by byte (like our ISO files) are identified by CRC-32 or SHA-1 hashes (or both) and even if the filesize is the same and the date stamp and all other identifiable metadata is the same as well, they are referred to as "statistically unique" if their hashes differ. The inversion of this is also true, with the files being known as "statistically identical" if their hashes are the same. Each hash has a certain inherent accuracy rate and when two or more different hashing algorithms are employed their rates are multiplied. So for example, with an ISO file like ours and identical SHA-1 and CRC-32 hashes the probability that two files are different approaches zero asymptotically and quickly disappears over the digital resolution horizon.
The files are the same.
)
