The specification is your friend. https://www.khronos.org/registry/cl/specs/opencl-1.2.pdf
. clBuildProgram must be called for program created using either clCreateProgramWithSource or clCreateProgramWithBinary to build the program executable for one or more devices associated with program.
So it's explicitly said that you must call the buildprogram even if you are using binary programs.
In most cases the buildprogram is faster for binaryprograms than what they are for text programs. So there is marginal speed benefit.