How do one-way hash functions work? (Edited)

Question

I read the Wikipedia article about md5 hashes but I still can't understand how a hash can't be "reconstituted" back to the original text.

Could someone explain to someone who knows very little about cryptography how this works? What part of the function makes it one-way?

Answer 1

Since everyone until now has simply defined what a hash function was, I will bite.

A one-way function is not just a hash function -- a function that loses information -- but a function f for which, given an image y ("SE" or 294 in existing answers), it is difficult to find a pre-image x such that f(x)=y.

This is why they are called one-way: you can compute an image but you can't find a pre-image for a given image.

None of the ordinary hash function proposed until now in existing answers have this property. None of them are one-way cryptographic hash functions. For instance, given "SE", you can easily pick up the input "SXXXE", an input with the property that X-encode("SXXXE")=SE.

There are no "simple" one-way functions. They have to mix their inputs so well that not only you don't recognize the input at all in the output, but you don't recognize another input either.

SHA-1 and MD5 used to be popular one-way functions but they are both nearly broken (specialist know how to create pre-images for given images, or are nearly able to do so). There is a contest underway to choose a new standard one, which will be named SHA-3.

An obvious approach to invert a one-way function would be to compute many images and keep them in a table associating to each image the pre-image that produced it. To make this impossible in practice, all one-way function have a large output, at least 64 bits but possibly much larger (up to, say, 512 bits).

EDIT: How do most cryptographic hash functions work?

Usually they have at their core a single function that does complicated transformations on a block of bits (a block cipher). The function should be nearly bijective (it shouldn't map too many sequences to the same image, because that would cause weaknesses later) but it doesn't have to be exactly bijective. And this function is iterated a fixed number of times, enough to make the input (or any possible input) impossible to recognize.

Take the example of Skein, one of the strong candidates for the SHA-3 context. Its core function is iterated 72 times. The only number of iterations for which the creators of the function know how to sometimes relate the outputs to some inputs is 25. They say it has a "safety factor" of 2.9.

Answer 2

Think of a really basic hash - for the input string, return the sum of the ASCII values of each character.

hash( 'abc' ) = ascii('a')+ascii('b')+ascii('c')
              = 97 + 98 + 99
              = 294

Now, given the hash value of 294, can you tell what the original string was? Obviously not, because 'abc' and 'cba' (and countless others) give the same hash value.

Cryptographic hash functions work the same way, except that obviously the algorithm is much more complex. There are always going to be collisions, but if you know string s hashes to h, then it should be very difficult ("computationally infeasible") to construct another string that also hashes to h.

Answer 3

Shooting for a simple analogy here instead of a complex explanation.

To start with, let's break the subject down into two parts, one-way operations and hashing. What is a one-way operation and why would you want one?

One way operations are called that because they are not reversible. Most typical operations like addition and multiplication can be reversed while modulo division can not be reversed. Why is that important? Because you want to provide a output value which 1) is difficult to duplicate without the original inputs and 2) provides no way to figure out the inputs from the output.

Reversible

Addition:

4 + 3 = 7  

This can be reversed by taking the sum and subtracting one of the addends

7 - 3 = 4  

Multiplication:

4 * 5 = 20  

This can be reversed by taking the product and dividing by one of the factors

20 / 4 = 5

Not Reversible

Modulo division:

22 % 7 = 1  

This can not be reversed because there is no operation that you can do to the quotient and the dividend to reconstitute the divisor (or vice versa).

Can you find an operation to fill in where the '?' is?

1  ?  7 = 22  
1  ?  22 = 7

With that being said, one-way hash functions have the same mathematical quality as modulo division.

Why is this important?

Lets say I gave you a key to a locker in a bus terminal that has one thousand lockers and asked you to deliver it to my banker. Being the smart guy you are, not to mention suspicious, you would immediately look on the key to see what locker number is written on the key. Knowing this, I've done a few devious things; first I found two numbers that when divided using modulo division gives me a number in the range between 1 and 1000, second I erased the original number and written on it the divisor from the pair of numbers, second I chose a bus terminal that has a guard protecting the lockers from miscreants by only letting people try one locker a day with their key, third the banker already knows the dividend so when he gets the key he can do the math and figure out the remainder and know which locker to open.

If I choose the operands wisely I can get near to a one-to-one relationship between the quotient and the dividend which forces you to try each locker because the answer spreads the results of the possible inputs over the range of desired numbers, the lockers available in the terminal. Basically, it means you can't acquire any knowledge about the remainder even if you know one of the operands.

So, now I can 'trust' you to deliver the key to its rightful owner without worrying that you can easily guess to which locker it belongs. Sure, you could brute force search all the lockers but that would take almost 3 years, plenty of time for my banker to use the key and empty the locker.

See the other answers for more specifics on the different hash functions.

Answer 4

Here's a very simple example. Assume that I'm a beginning cryptographer and I create a hash function that does the following:

int SimpleHash(file) {
    return 0 if file.length is even;
    return 1 if file.length is odd;
}

Now here's the test. SimpleHash(specialFile) is 0. What was my original file?

Obviously, there's no way to know (although you could likely discover pretty easily that my hash is based on file length). There is no way to "reconstitute" my file based on the hash because the hash doesn't contain everything that my file did.

Answer 5

A hash is a (very) lossy encoding.

To give you a simpler example, imagine a fictitious 2-letter encoding of a 5-letter word called the X-encoding. The algorithm for the X-encoding is simple: take the first and last letters of the word.

So,

X-encode( SAUCE ) = SE
X-encode( BLOCK ) = BK

Clearly, you cannot reconstruct SAUCE from its encoding SE (assuming our range of possible inputs is all 5-letter words). The word could just as easily be SPACE.

As an aside, the fact that SAUCE and SPACE both produce SE as an encoding is called a collision, and you can see that the X-ecoding wouldn't make a very good hash. :)

Answer 6

In simple terms, a hash function works by making a big tangled mess of the input data.

See MD5 for instance. It processes input data by 512-bit blocks. Each block is split into 16 32-bit words. There are 64 steps, each step using one of the 16 input words. So each word is used four times within the course of the algorithm. This is where one-wayness comes from: any input bit is input at several places, and between two such inputs the function mixes all the current data together so that each input bit impacts most of the 128-bit running state. This prevents you from inverting the function, or computing a collision, by looking at only a part of the data. You have to look at the whole 128 bits, and the space of 128-bit blocks is too wide to be efficiently walked through.

Now MD5 does not do a good job at it, since collisions for that function can be found. From a cryptographer point of view, MD5 is a rotated encryption function. The processing of one message block M (512 bits) uses an input state V (a 128-bit value) and computes the new state V' as V' = V + E(M, V) where '+' is a word-wise addition, and 'E' happens to be a symmetric encryption function (aka a 'block cipher') which uses M as key and V as the message to be encrypted. From a closer look, E can is a kind of "extended Feistel network", similar to the DES block cipher, with four quarters instead of two halves. Details are not important here; my point is that what makes a "good" hash function, among hash functions which use that structure (called "Merkle-Damgård"), is similar to what makes a block cipher "secure". The successful collision attacks on MD5 use differential cryptanalysis, a tool which was designed to attack block ciphers in the first place.

From a good block cipher to a good hash function, there is a step which is not to be dismissed. With the Merkle-Damgård structure, the hash function is secure if the underlying block cipher is resistant to "related key attacks", a rather obscure property against which block ciphers are rarely strengthened because, for symmetric encryption, related key attacks barely have any practical impact. For instance, the AES encryption turned out not to be as resistant to related key attacks as could be wished for, and this did not trigger general panic. That resistance was not part of the properties which were sought for when AES was designed. It just prevents turning the AES into a hash function. There is a hash function called Whirlpool, which builds on a derivate of Rijndael, "Rijndael" being the initial name of what became the AES; but Whirlpool takes care to modify the parts of Rijndael which are weak to related key attacks.

Also, there are other structures which can be used for building a hash function. The current standard functions (MD5, SHA-1, and the "SHA-2" family, aka SHA-224, SHA-256, SHA-384 and SHA-512) are Merkle-Damgård functions, but many of the would-be successors are not. There is an ongoing competition, organized by the NIST (the US federal organization which deals with that kind of things), to select a new standard hash function, dubbed "SHA-3". See this page for details. Right now, they are down to 14 candidates from an initial 51 (not counting a dozen extra which failed the administrative test of sending a complete submission with code which compiles and runs properly).

Let's now have a more conceptual look. A secure hash function should look like a random oracle: an oracle is a black box which, when given a message M as input, outputs an answer h(M) which is chosen at random, uniformly, in the output space (i.e. all n-bit strings if the hash function output length is n). If given the same message M again as input, the oracle outputs the same value than previously. Apart from that restriction, the output of the oracle on a non previously used input M is unpredictable. One can imagine the oracle as a container for a gnome who throws dice, and carefully records the input messages and corresponding outputs in a big book, so that he will honor his oracle contract. There is no way to predict what the next output will be since the gnome himself does not know that.

If a random oracle exists, then inverting the hash function has cost 2^n: in order to have a given output, there is no better strategy than using distinct input messages until one yields the expected value. Due to the uniform random selection, probability of success is 1/(2^n) at each try, and the average number of requests to the dice-throwing gnome will be 2^n. For collisions (finding two distinct inputs which yields the same hash value), the cost is about *1.4*2^(n/2)* (roughly speaking, with *1.4*2^(n/2)* outputs, we can assemble about 2^n pairs of output, each having a probability of 1/(2^n) of matching, i.e. having two distinct inputs which have the same output). These are the best that can be done with a random oracle.

Therefore, we look for hash functions which are as good as a random oracle: they must mix the input data in such a way that we cannot find a collision more efficiently than what it would cost to simply invoke the function 2^(n/2) times. The bane of hash function is mathematical structure, i.e. shortcuts which allow the attacker to view the hash function internal state (which is big, at least n bits) as a variation on a mathematical object which lives in a much shorter space. 30 years of public research on symmetric encryption systems have produced a whole paraphernalia of notions and tools (diffusion, avalanche, differentials, linearity...) which can be applied. Bottom-line, however, is that we have no proof that a random oracle may actually exist. We want a hash function which cannot be attacked. What we have are hash function candidates, for which no attack is currently known, and, somewhat better, we have some functions for which some kinds of attack can be proven not to work.

There is still some research to be done.

Answer 7

array
With some squinting, associative arrays look very much like hashes. The major differences were the lack of the % symbol on hash names, and that one could only assign to them one key at a time. Thus, one would say $foo{'key'} = 1;, but only @keys = keys(foo);. Familiar functions like each, keys, and values worked as they do now (and delete was added in Perl 2).

Perl 3 had three whole data types: it had the % symbol on hash names, allowed an entire hash to be assigned to at once, and added dbmopen (now deprecated in favour of tie). Perl 4 used comma-separated hash keys to emulate multidimensional arrays (which are now better handled with array references).

Perl 5 took the giant leap of referring to associative arrays as hashes. (As far as I know, it is the first language to have referred to the data structure thus, rather than "hash table" or something similar.) Somewhat ironically, it also moved the relevant code from hash.c into hv.c.

Nomenclature
Dictionaries, as explained earlier, are unordered collections of values indexed by unique keys. They are sometimes called associative arrays or maps. They can be implemented in several ways, one of which is by using a data structure known as a hash table (and this is what Perl refers to as a hash).

Perl's use of the term "hash" is the source of some potential confusion, because the output of a hashing function is also sometimes called a hash (especially in cryptographic contexts), and because hash tables aren't usually called hashes anywhere else.

To be on the safe side, refer to the data structure as a hash table, and use the term "hash" only in obvious, Perl-specific contexts.