This post explains the issue nicely. http://www.hackerfactor.com/blog/index.php?/archives/307-Picture-Go-Back.html
[1] comment for AVStream structure clearly mentions that "r_frame_rate" is a guess and may not be accurate, because even if I have frame-rate of (say) 25fps, in term of base_time I may have 24 or 26 frames in a second.
[2] To find the exact frame number you need to decode frame from the start and keep a counter, but that is very in-efficient, this can be optimized for some file-formats like MP4 where information about every frame is present in file-header.