Opencv - detecting whether the eye is closed or open

Question

I am working on a project where we are trying to detect whether the eye is closed or open in a picture. What we have done so far is that we detected the face, then the eyes. Then we applied hough transform, hoping that the iris would be the only circle when the eye is open. The problem is that when the eye is closed, it produces a circle as well:

Here is the code:

import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.core.MatOfRect;
import org.opencv.core.Point;
import org.opencv.core.Rect;
import org.opencv.core.Scalar;
import org.opencv.core.Size;
import org.opencv.highgui.Highgui;
import org.opencv.objdetect.CascadeClassifier;
import org.opencv.imgproc.Imgproc;



 
public class FaceDetector {
 
    public static void main(String[] args) {
        
    
 
        System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
        System.out.println("\nRunning FaceDetector");
 
        CascadeClassifier faceDetector = new CascadeClassifier("D:\\CS\\opencv\\sources\\data\\haarcascades\\haarcascade_frontalface_alt.xml");
        CascadeClassifier eyeDetector = new CascadeClassifier("D:\\CS\\opencv\\sources\\data\\haarcascades\\haarcascade_eye.xml");
       
        Mat image = Highgui.imread("C:\\Users\\Yousra\\Desktop\\images.jpg");
        Mat gray = Highgui.imread("C:\\Users\\Yousra\\Desktop\\eyes\\E7.png");
        
        String faces;
        String eyes;
        
 
        MatOfRect faceDetections = new MatOfRect();
        MatOfRect eyeDetections = new MatOfRect();
        
        Mat face;
        Mat crop = null;
        Mat circles = new Mat();
        faceDetector.detectMultiScale(image, faceDetections);
        
   for (int i = 0; i< faceDetections.toArray().length; i++){
            
            faces = "Face"+i+".png";
            
             face = image.submat(faceDetections.toArray()[i]);
             crop = face.submat(4, (2*face.width())/3, 0, face.height());
            Highgui.imwrite(faces, face);
             eyeDetector.detectMultiScale(crop, eyeDetections, 1.1, 2, 0,new Size(30,30), new Size()); 
         
             if(eyeDetections.toArray().length ==0){
                 
                 System.out.println(" Not a face" + i);
             }else{
                 
                 System.out.println("Face with " + eyeDetections.toArray().length + "eyes" );
                 
                 for (int j = 0; j< eyeDetections.toArray().length ; j++){
                     
                    System.out.println("Eye" );
                    Mat eye = crop.submat(eyeDetections.toArray()[j]);
                    eyes = "Eye"+j+".png";
                    Highgui.imwrite(eyes, eye);
                     
                 }
             }
         }
             
             
          
        
            
             Imgproc.cvtColor(gray, gray, Imgproc.COLOR_BGR2GRAY);
             System.out.println("1 Hough :" +circles.size());
 float circle[] = new float[3];

             for (int i = 0; i < circles.cols(); i++)
             {
                     circles.get(0, i, circle);
                 org.opencv.core.Point center = new org.opencv.core.Point();
                 center.x = circle[0];
                 center.y = circle[1];
                 Core.circle(gray, center, (int) circle[2], new Scalar(255,255,100,1), 4);
                 }
             
             
             Imgproc.Canny( gray, gray, 200, 10, 3,false);  
             
             Imgproc.HoughCircles( gray, circles, Imgproc.CV_HOUGH_GRADIENT, 1, 100, 80, 10, 10, 50 );
             System.out.println("2 Hough:" +circles.size());
             
             for (int i = 0; i < circles.cols(); i++)
             {
                     circles.get(0, i, circle);
                 org.opencv.core.Point center = new org.opencv.core.Point();
                 center.x = circle[0];
                 center.y = circle[1];
                 Core.circle(gray, center, (int) circle[2], new Scalar(255,255,100,1), 4);
                 }
             Imgproc.Canny( gray, gray, 200, 10, 3,false);  
             
             Imgproc.HoughCircles( gray, circles, Imgproc.CV_HOUGH_GRADIENT, 1, 100, 80, 10, 10, 50 );
             System.out.println("3 Hough" +circles.size());
             
             //float circle[] = new float[3];

             for (int i = 0; i < circles.cols(); i++)
             {
                     circles.get(0, i, circle);
                 org.opencv.core.Point center = new org.opencv.core.Point();
                 center.x = circle[0];
                 center.y = circle[1];
                 Core.circle(gray, center, (int) circle[2], new Scalar(255,255,100,1), 4);
                 }

            String hough = "afterhough.png";
            Highgui.imwrite(hough, gray);
   }     
}

How to make it more accurate?

Answer 1

Circular Hough transform is unlikely to work well in the majority of cases i.e. where the eye is partially open or closed. You'd be better off isolating rectangular regions (bounding boxes) around the eyes and computing a measure based on pixel intensities (grey levels). For example the variance of pixels within the region would be a good discriminator between open and closed eyes. Obtaining a bounding box around the eyes can be done quite reliably using relative position from the bounding box detected around the face using OpenCV Haar cascades. Figure 3 in this paper gives some idea of the location process.

http://personal.ee.surrey.ac.uk/Personal/J.Collomosse/pubs/Malleson-IJCV-2012.pdf

Answer 2

You can check circles.cols() value if it is 2 then the eyes are open and if the value is 0 then the eyes are closed. You can also detect the blinking of eye if the value of circles.cols() changes from 2 to 0. Hough transform wil not detect a circle if the eyes are closed.