Confused about x264 and encoding video frames

Question

I built a test driver for encoding a series of images I have captured. I am using libx264 and based my driver off of this guy's answer:

StackOverflow link

In my case I am starting out by reading in a JPG image and converting to YUV and passing that same frame over and over in a loop to the x264 encoder.

My expectation was that since the frame is the same that the output from the encoder would be very small and constant.

Instead I find that the NAL payload is varied from a few bytes to a few KB and also varies highly depending on the frame rate I specify in the encoder parameters.

Obviously I don't understand video encoding. Why does the output size vary so much?

int main() 
{
  Image image(WIDTH, HEIGHT);
  image.FromJpeg("frame-1.jpg");

  unsigned char *data = image.GetRGB();

  x264_param_t param;

  x264_param_default_preset(&param, "fast", "zerolatency");
  param.i_threads = 1;
  param.i_width = WIDTH;
  param.i_height = HEIGHT;
  param.i_fps_num = FPS;
  param.i_fps_den = 1;

  // Intra refres:
  param.i_keyint_max = FPS;
  param.b_intra_refresh = 1;

  //Rate control:
  param.rc.i_rc_method = X264_RC_CRF;
  param.rc.f_rf_constant = FPS-5;
  param.rc.f_rf_constant_max = FPS+5;

  //For streaming:
  param.b_repeat_headers = 1;
  param.b_annexb = 1;

  x264_param_apply_profile(&param, "baseline");

  // initialize the encoder
  x264_t* encoder = x264_encoder_open(&param);
  x264_picture_t pic_in, pic_out;
  x264_picture_alloc(&pic_in, X264_CSP_I420, WIDTH, HEIGHT);
  // X264 expects YUV420P data use libswscale 
  // (from ffmpeg) to convert images to the right format
  struct SwsContext* convertCtx =
        sws_getContext(WIDTH, HEIGHT, PIX_FMT_RGB24, WIDTH, HEIGHT,
                       PIX_FMT_YUV420P, SWS_FAST_BILINEAR,
                       NULL, NULL, NULL);

  // encoding is as simple as this then, for each frame do:
  // data is a pointer to your RGB structure
  int srcstride = WIDTH*3; //RGB stride is just 3*width
  sws_scale(convertCtx, &data, &srcstride, 0, HEIGHT,
            pic_in.img.plane, pic_in.img.i_stride);
  x264_nal_t* nals;
  int i_nals;
  int frame_size =
        x264_encoder_encode(encoder, &nals, &i_nals, &pic_in, &pic_out);

  int max_loop=15;
  int this_loop=1;

  while (frame_size >= 0 && --max_loop)
  {
      cout << "------------" << this_loop++ << "-----------------\n";
      cout << "Frame size = " << frame_size << endl;
      cout << "output has " << pic_out.img.i_csp << " colorspace\n";
      cout << "output has " << pic_out.img.i_plane << " # img planes\n";

      cout << "i_nals = " << i_nals << endl;
      for (int n=0; n<i_nals; n++)
        cout << "nal[" << n << "] payload size = "
             << nals[n].i_payload << endl;



      // clean this input frame and grab another
      x264_picture_clean(&pic_in);
      x264_picture_alloc(&pic_in, X264_CSP_I420, WIDTH, HEIGHT);
      sws_scale(convertCtx, &data, &srcstride, 0, HEIGHT,
                pic_in.img.plane, pic_in.img.i_stride);


      frame_size=
        x264_encoder_encode(encoder, &nals, &i_nals, &pic_in, &pic_out);
  }

  delete [] data;
  return 0;
}

My original JPG image that I'm reading in is 320x240 and is roughly 9KB on disk. The output from running that same image in through the encoder looks like this:

x264 [warning]: ref > 1 + intra-refresh is not supported
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2 AVX
x264 [info]: profile Constrained Baseline, level 1.3 
------------1-----------------
Frame size = 17412
output has 3 colorspace
output has 2 # img planes
i_nals = 4 
nal[0] payload size = 26
nal[1] payload size = 8 
nal[2] payload size = 610 
nal[3] payload size = 16768
------------2-----------------
Frame size = 11
output has 3 colorspace
output has 2 # img planes
i_nals = 1 
nal[0] payload size = 11
------------3-----------------
Frame size = 11
output has 3 colorspace
output has 2 # img planes
i_nals = 1 
nal[0] payload size = 11
------------4-----------------
Frame size = 16
output has 3 colorspace
output has 2 # img planes
i_nals = 1 
nal[0] payload size = 16
------------5-----------------
Frame size = 104 
output has 3 colorspace
output has 2 # img planes
i_nals = 1 
nal[0] payload size = 104 
------------6-----------------
Frame size = 973 
output has 3 colorspace
output has 2 # img planes
i_nals = 1
nal[0] payload size = 973
------------7-----------------
Frame size = 4576
output has 3 colorspace
output has 2 # img planes
i_nals = 1
nal[0] payload size = 4576
------------8-----------------
Frame size = 4040
output has 3 colorspace
output has 2 # img planes
i_nals = 1
nal[0] payload size = 4040
------------9-----------------
Frame size = 3917
output has 3 colorspace
output has 2 # img planes
i_nals = 1
nal[0] payload size = 3917
------------10-----------------
Frame size = 3799
output has 3 colorspace
output has 2 # img planes
i_nals = 1
nal[0] payload size = 3799
------------11-----------------
Frame size = 2454
output has 3 colorspace
output has 2 # img planes
i_nals = 1
nal[0] payload size = 2454
------------12-----------------
Frame size = 1641
output has 3 colorspace
output has 3 colorspace
output has 2 # img planes
i_nals = 1
nal[0] payload size = 1641
------------13-----------------
Frame size = 1121
output has 3 colorspace
output has 2 # img planes
i_nals = 1
nal[0] payload size = 1121
------------14-----------------
Frame size = 247
output has 3 colorspace
output has 2 # img planes
i_nals = 1
nal[0] payload size = 247

If I set FPS to 10 instead of 30 then the output size changes dramatically:

x264 [warning]: ref > 1 + intra-refresh is not supported
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2 AVX
x264 [info]: profile Constrained Baseline, level 1.1 
------------1-----------------
Frame size = 60120
output has 3 colorspace
output has 2 # img planes
i_nals = 4 
nal[0] payload size = 25
nal[1] payload size = 9 
nal[2] payload size = 609 
nal[3] payload size = 59477
------------2-----------------
Frame size = 10
output has 3 colorspace
output has 2 # img planes
i_nals = 1 
nal[0] payload size = 10
------------3-----------------
Frame size = 151 
output has 3 colorspace
output has 2 # img planes
i_nals = 1 
nal[0] payload size = 151 
------------4-----------------
Frame size = 2329
output has 3 colorspace
output has 2 # img planes
i_nals = 1 
nal[0] payload size = 2329
------------5-----------------
Frame size = 1466
output has 3 colorspace
output has 2 # img planes
i_nals = 1 
nal[0] payload size = 1466
------------6-----------------
Frame size = 539 
output has 3 colorspace
output has 2 # img planes
i_nals = 1
nal[0] payload size = 539
------------7-----------------
Frame size = 311
output has 3 colorspace
output has 2 # img planes
i_nals = 1
nal[0] payload size = 311
------------8-----------------
Frame size = 291
output has 3 colorspace
output has 2 # img planes
i_nals = 1
nal[0] payload size = 291
------------9-----------------
Frame size = 302
output has 3 colorspace
output has 2 # img planes
i_nals = 1
nal[0] payload size = 302
------------10-----------------
Frame size = 287
output has 3 colorspace
output has 2 # img planes
i_nals = 1
nal[0] payload size = 287
------------11-----------------
Frame size = 11937
output has 3 colorspace
output has 2 # img planes
i_nals = 4
nal[0] payload size = 25
nal[1] payload size = 9
nal[2] payload size = 9
nal[3] payload size = 11894

Can you enlighten me?

Answer 1

Rhe issue is mainly due "rate control" parameter. Remove CRF rate control and use constant quantizer. Constant quantizer is not a recommended mode of operation but in your case it (same image repetition) would produce similar size frames.

Remove:

param.b_intra_refresh = 1;

//Rate control:
param.rc.i_rc_method = X264_RC_CRF;
param.rc.f_rf_constant = FPS-5;
param.rc.f_rf_constant_max = FPS+5;

Then Add:

 param.rc.i_qp_constant = 18; 
 param.rc.i_qp_min = 18; 
 param.rc.i_qp_max = 18;