SVG!=CSS. Once you set your viewbox units in SVG, those units usually override any units specified in the rest of the document (like those "px" in your <image>
). SVG is reading those values of "500px" and as 500 ViewBox Units (which in your case are 1/600th's of a pixel (1636/623316), then applying the transform specified which scales it by 381x and moves it by 6096 viewbox units (aka about 10pixels) in x and y.
But your real problem is that the clipPath units are inheriting that transform (e.g. those clip units are being interpreted as units in a (623,316 x 381) width container!!)
If you redo your image dimensions using viewbox units, and get rid of the transform everything works. The following is only roughly approximate, you need to get your calculator out to get the real results - but it makes the point.
<image x="0" y="0" height="323000" width="150000"
xlink:href="http://www.cutepandapictures.com/wp-
content/uploads/2012/07/babypandahugsatree.jpg" clip-path="url(#randompath)"/>
Alternatively, divide all your clip path values by 500'ish, and you'll see a result that starts to make sense. Here's an example with the clip path numbers divided by 1,000 (it was easy to just move the decimal point): http://jsfiddle.net/reU7t/