Quick answer: no, this isn't possible using just OpenGL calls. GL_SRC_ALPHA_SATURATE
shouldn't do what you want either, its RGB factor would be min(Asrc, (1-Adest))
, in your case that would probably always end up being equal to Asrc
, as I assume your destination alpha is always 0.
Long Answer: Your best and only option is to write said fragment shader. In order to achieve what you want you'll have to use some trickery. I recommend pre-multiplying your overlay color with an alpha factor and output another alpha value to blend the destination color. An equation similar to this should suffice:
// Note I cap off the alpha values in both cases to achieve the desired non-linear ramp
float alpha = overlay_opacity * 2.0f;
out = vec4(color.rgb * min(1.0f, alpha), min(1.0f, 2.0f - alpha));
Then use the following blend functions to make sure the destination color blends as well:
glBlendFunc(GL_ONE, GL_SRC_ALPHA);
One further optimization, you notice I multiply the overlay opacity by two to cap it off for both alpha values. Instead of that you can pre multiply it by two before passing it to the shader, this will shave off some multiplications for each pixel.
Hopefully this will solve your problem, good luck!