Android OpenCV MatToBitmap causes black image

Question

I am using OpenCV on android. I try to detect largest rectangle and extract this rectangle from image. I am using this code below. When the photo is taken, some image processing is done. My aim is, I want to detect a rectangle which is like border of paper and if its area is bigger than threshold_area, I want to crop it from original bitmap and draw on an imageview.

Mat cropped_mat;

PictureCallback _jpgCallBack = new PictureCallback(){
    @Override
    public void onPictureTaken(byte[] data, Camera camera) {

        mCamera.startPreview();

        bitmap = BitmapFactory.decodeByteArray(data , 0, data .length); 
        Mat ImageMat = new Mat();
        Utils.bitmapToMat(cropped_mat, ImageMat); 
        double area = findLargestRectangle(ImageMat);

        if(area > THRESHOLD_AREA){

            mCamera.stopPreview();

            try{
                bitmap=Bitmap.createBitmap(cropped_mat.cols(), cropped_mat.rows(),Bitmap.Config.ARGB_8888);
                Utils.matToBitmap(cropped_mat, bitmap);
            }
            catch(Exception e){}

            img.setImageBitmap(bitmap);

        }
        else{
            takePicture();
        }

    }
};

This is my findLargestRectangle method.

private double findLargestRectangle(Mat imgSource) {


    Imgproc.Canny(imgSource, imgSource, 50, 50);


    Imgproc.GaussianBlur(imgSource, imgSource, new Size(5, 5), 5);


    List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
    Imgproc.findContours(imgSource, contours, new Mat(), Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);

    double maxArea = -1;
    int maxAreaIdx = -1;        

    MatOfPoint temp_contour = contours.get(0); //the largest is at the index 0 for starting point
    MatOfPoint2f approxCurve = new MatOfPoint2f();
    Mat largest_contour = contours.get(0);

    List<MatOfPoint> largest_contours = new ArrayList<MatOfPoint>();
    for (int idx = 0; idx < contours.size(); idx++) {

        temp_contour = contours.get(idx);
        double contourarea = Imgproc.contourArea(temp_contour);

        //compare this contour to the previous largest contour found
        if (contourarea > maxArea) {
            //check if this contour is a square
            MatOfPoint2f new_mat = new MatOfPoint2f( temp_contour.toArray() );
            int contourSize = (int)temp_contour.total();
            Imgproc.approxPolyDP(new_mat, approxCurve, contourSize*0.05, true);

            if (approxCurve.total() == 4) {
                maxArea = contourarea;
                maxAreaIdx = idx;
                largest_contours.add(temp_contour);
                largest_contour = temp_contour;
            }
        }
    }

    MatOfPoint temp_largest = largest_contours.get(largest_contours.size()-1);
    Rect rect = Imgproc.boundingRect(temp_largest);

    largest_contours = new ArrayList<MatOfPoint>();
    largest_contours.add(temp_largest);

    Core.rectangle(imgSource, new Point(rect.x,rect.y), new Point(rect.x+rect.width,rect.y+rect.height),new Scalar(0,0,255));
    cropped_mat= imgSource.submat(rect.y , rect.y + rect.height, rect.x, rect.x + rect.width);
    return Imgproc.contourArea(temp_largest);
}

But result is like below. Where am i wrong ?

original frame

result frame

intended bitmap

Answer 1

The reference to the image object you are processing within onPictureTaken is being passed by value into findLargestRectangle.

You are then manipulating that image with destructive operations specifically: 1) Canny edge detection, which replaces the raw image content with a binary mask of edges present in the image; 2) a Blur; 3) A contour finding operation, which is also destructive as this OpenCV function "eats away" contours as it discovers and traverses them.

The result is a Canny edge-like image with many of its edges missing. That is exactly what your output is displaying when you subsequently update your control "img" with the content of the image.

You can fix this by making a local copy of the image object inside findLargestRectangle and processing that copy, to prevent this corruption.