I fixed this by adding the --enable-memalign-hack
to the configure flags for ffmpeg. I narrowed the problem down to: libavutil/mem.c
which includes some preprocessor defines for various memory handlers which led me to that configure flag.
Not sure if this is the right fix, or whether I'm setting myself up for problems later. Looks like someone else has had similar problems here: http://ffmpeg.org/pipermail/ffmpeg-devel/2013-February/138634.html