I don't think there is a way to set the capture area in AVCaptureConnection. Instead you could try to read the image off the screen and then crop it to your requirement.
If UIImage *photoImage is the image being returned on capturing the photo
CGRect refRect; //Define this to the exact frame which you want to crop the larger image to i.e. with smaller frame.size.height
CGFloat deviceScale = photoImage.scale;
CGImageRef imageRef = CGImageCreateWithImageInRect(photoImage.CGImage. refRect);
UIIImage *finalPhoto = [[UIImage alloc] initWithCGImage:imageRef deviceScale orientation:photoImage.imageOrientation];
The finalPhoto will now be cropped to what you want