There is at least two bugs in your implementation.
- The indexing by the brightness because of using a temporary of type signed char.
- The final normalization result can be influence by the value of control height and the maximum count of pixel in a bin. The division cannot really be put outside of the loop because of that.
I recommend also to use a std::array
( need c++11 ) to store the result instead of a raw pointer as there is a risk the caller do not allocate enough space for what will use the function.
#include <algorithm>
#include <array>
void calculateHistogram(const unsigned char* yuv420sp, const int yuvWidth, const int yuvHeight, const int histogramControlHeight, std::array<int, 256> &outHistogramData ) {
outHistogramData.fill(0);
std::for_each( yuv420sp, yuv420sp + yuvWidth * yuvHeight, [&](int e) {
outHistogramData[e]++;
} );
int maxCountInBins = * std::max_element( begin(outHistogramData), end(outHistogramData) );
for( int &bin : outHistogramData )
bin = bin * histogramControlHeight / maxCountInBins;
}