It is not QCustomPlot related question actually.
You must fill space between points with some splines. Here is a nice article that shows what splines are: http://cairnarvon.rotahall.org/2009/07/05/quadratic-spline-interpolation/
So basically you need two QCPGraph
's.
First graph should hold your key points without any connection lines (QCPGraph::LineStyle::lsNone
).
Second graph should hold calculated (by your code!) spline points with some resolution (for example, spline points for minX, minX+dx, minX+dx*2 .. maxX) and without scatters.
So you will get your points connected to each other with sexy curves.