Just got the answer from the gstreamer mailing list. In case anyone else is having the same problem, adding the gstrtpjitterbuffer element fixes it.
gst-launch-0.10 udpsrc port=5000 \
caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, sprop-parameter-sets=(string)\"J01AHqkYGwe83gDUBAQG2wrXvfAQ\\,KN4JyA\\=\\=\", payload=(int)96, ssrc=(uint)786848209, clock-base=(uint)101553131, seqnum-base=(uint)64602" \
! gstrtpjitterbuffer latency=1000
! rtph264depay ! ffdec_h264 ! ffmpegcolorspace ! autovideosink