Converting Camera Coordinates to Custom View Coordinates

Question

I am trying to make a simple face detection app consisting of a SurfaceView (essentially a camera preview) and a custom View (for drawing purposes) stacked on top. The two views are essentially the same size, stacked on one another in a RelativeLayout. When a person's face is detected, I want to draw a white rectangle on the custom View around their face.

The Camera.Face.rect object returns the face bound coordinates using the coordinate system explained here and the custom View uses the coordinate system described in the answer to this question. Some sort of conversion is needed before I can use it to draw on the canvas.

Therefore, I wrote an additional method ScaleFacetoView() in my custom view class (below) I redraw the custom view every time a face is detected by overriding the OnFaceDetection() method. The result is the white box appears correctly when a face is in the center. The problem I noticed is that it does not correct track my face when it moves to other parts of the screen.

Namely, if I move my face:

• Up - the box goes left
• Down - the box goes right
• Right - the box goes upwards
• Left - the box goes down

I seem to have incorrectly mapped the values when scaling the coordinates. Android docs provide this method of converting using a matrix, but it is rather confusing and I have no idea what it is doing. Can anyone provide some code on the correct way of converting Camera.Face coordinates to View coordinates?

Here's the code for my ScaleFacetoView() method.

public void ScaleFacetoView(Face[] data, int width, int height, TextView a){

     //Extract data from the face object and accounts for the 1000 value offset 
     mLeft = data[0].rect.left + 1000;
     mRight = data[0].rect.right + 1000;
     mTop = data[0].rect.top + 1000;
     mBottom = data[0].rect.bottom + 1000;

     //Compute the scale factors
     float xScaleFactor = 1;
     float yScaleFactor = 1;

     if (height > width){
         xScaleFactor = (float) width/2000.0f;
         yScaleFactor = (float) height/2000.0f;          
     }
     else if (height < width){
         xScaleFactor = (float) height/2000.0f;
         yScaleFactor = (float) width/2000.0f;
     }

     //Scale the face parameters
     mLeft = mLeft * xScaleFactor; //X-coordinate
     mRight = mRight * xScaleFactor; //X-coordinate
     mTop = mTop * yScaleFactor; //Y-coordinate
     mBottom = mBottom * yScaleFactor; //Y-coordinate

}

As mentioned above, I call the custom view like so:

@Override
public void onFaceDetection(Face[] arg0, Camera arg1) {
    if(arg0.length == 1){

        //Get aspect ratio of the screen
        View parent = (View) mRectangleView.getParent();            
        int width = parent.getWidth();
        int height = parent.getHeight();

        //Modify xy values in the view object
        mRectangleView.ScaleFacetoView(arg0, width, height);
        mRectangleView.setInvalidate();
        //Toast.makeText( cc ,"Redrew the face.", Toast.LENGTH_SHORT).show();
        mRectangleView.setVisibility(View.VISIBLE);
  //rest of code

Answer 1

The Camera.Face class returns the face bound coordinates using the image frame that the phone would save into its internal storage, rather than using the image displayed in the Camera Preview. In my case, the images were saved in a different manner from the camera, resulting in a incorrect mapping. I had to manually account for the discrepancy by taking the coordinates, rotating it counter clockwise 90 degrees and flipping it on the y-axis prior to scaling it to the canvas used for the custom view.

EDIT: It would also appear that you can't change the way the face bound coordinates are returned by modifying the camera capture orientation using the Camera.Parameters.setRotation(int) method either.

Answer 2

Using the explanation Kenny gave I manage to do the following.

This example works using the front facing camera.

RectF rectF = new RectF(face.rect);
Matrix matrix = new Matrix();
matrix.setScale(1, 1);
matrix.postScale(view.getWidth() / 2000f, view.getHeight() / 2000f);
matrix.postTranslate(view.getWidth() / 2f, view.getHeight() / 2f);
matrix.mapRect(rectF);

The returned Rectangle by the matrix has all the right coordinates to draw into the canvas.

If you are using the back camera I think is just a matter of changing the scale to:

matrix.setScale(-1, 1);

But I haven't tried that.