That image is not formed of raw bytes - rather it is an encoded JPEG file. Moreover, you are not parsing the ascii HEX representation of the stream into proper bytes: that is, an "ff" sequence in that file is being passed to PIL as two c letters "f" instead of a byte with the number 255.
So, first, you decode the string to a proper byte sequence - sorry for the convolution - it is likely there is a better way to do that, but I am on a slow day:
data = urllib.urlopen("http://pastebin.ca/raw/2311595").read()
r_data = "".join(chr(int(data[i:i+2],16)) for i in range(0, len(data),2))
Ah, C.Y. posted on hte comment - this way:
>>> import binascii
>>> b_data = binascii.unhexlify(data)
And now, you import it to PIL as a JPEG image:
>>> from PIL import Image
>>> import cStringIO as StringIO
>>> stream = StringIO.StringIO(b_data)
>>> img = Image.open(stream)
>>> img.size
(320, 240)