MD5 is a cryptographic hash function, as such it intentionally produces a vastly different hash if even a single bit changes in its input. Therefore, just because two inputs are "close" (i.e. have few differences), the MD5 hashes will not be, that's inherent in the algorithm, and can't be "tuned" in the way you require.
There are many methods to detect similar images. One possibility is given here: http://www.hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html
The basic method given is to convert the images being compared to grayscale, resize them so that they are only a few pixels wide/tall, and then convert each pixel to black/white (1 bit/pixel) by comparing it to the average value of all pixels. Then you can compute pixel differences between the two images - the greater the number of different pixels, the greater the likely difference in the original images.
This method can be tuned by changing how much you reduce the original image - the more you reduce it, the less sensitive it'll be to differences between the images.