I assume you have configured swscale to YUV420p color space. 420P means 4:2:0 planar. Planar means the color channels are separate.
The luminance data (Y) is stored in the buffer point to pFrame->data[0] (Cb and Cr are in pFrame->data[1] and pFrame->data[2] respectively). in YUV420 the Y plane is 1 byte per pixel.
hence:
uint8_t getY(int x, int y, AVFrame *f)
{
return f->data[0][(y*f->linesize)+x];
}