I got a response from the ffmpeg bugtracker:
https://ffmpeg.org/trac/ffmpeg/ticket/2074
I suspect your are missing the codec delay. The video frame you get out usually isn't based on the packet you just sent in at all. This is especially true with multithreaded decoding, but even without it's not generally possible due to how B-frames work.
And, indeed I print out the ->pict_type "picture type" 3 frames later, there is an I-frame that pops up a few frames later with a "delay" of sorts exactly a 3 frames later. In my case, I have a quad-core computer, so perhaps there's a thread being allocated for each core on my computer to do the encoding.