Why does AVCaptureVideoOrientation landscape modes result in upside down still images?

Question

I am using AVFoundation classes to implement a custom camera in my app. I am only capturing still images, not video. I have everything working but am stumped by something. I take into account the device orientation when a still image is captured and set the videoOrientation of the video connection appropriately. A code snippet:

    // set the videoOrientation based on the device orientation to
    // ensure the pic is right side up for all orientations
    AVCaptureVideoOrientation videoOrientation;
    switch ([UIDevice currentDevice].orientation) {
        case UIDeviceOrientationLandscapeLeft:
            // Not clear why but the landscape orientations are reversed
            // if I use AVCaptureVideoOrientationLandscapeLeft here the pic ends up upside down
            videoOrientation = AVCaptureVideoOrientationLandscapeRight;
            break;
        case UIDeviceOrientationLandscapeRight:
            // Not clear why but the landscape orientations are reversed
            // if I use AVCaptureVideoOrientationLandscapeRight here the pic ends up upside down
            videoOrientation = AVCaptureVideoOrientationLandscapeLeft;
            break;
        case UIDeviceOrientationPortraitUpsideDown:
            videoOrientation = AVCaptureVideoOrientationPortraitUpsideDown;
            break;
        default:
            videoOrientation = AVCaptureVideoOrientationPortrait;
            break;
    }

    videoConnection.videoOrientation = videoOrientation;

Note my comments in the landscape cases. I have to reverse the orientation mapping or the resulting image is upside down. I capture and save the image with the following code:

[self.stillImageOutput captureStillImageAsynchronouslyFromConnection:videoConnection 
    completionHandler:^(CMSampleBufferRef imageSampleBuffer, NSError *error)
    {
        NSData *imageData = [AVCaptureStillImageOutput jpegStillImageNSDataRepresentation:imageSampleBuffer];
        self.stillImage = [UIImage imageWithData:imageData];
        // notify observers (image gets saved to the camera roll)                                                           
        [[NSNotificationCenter defaultCenter] postNotificationName:CaptureSessionManagerDidCaptureStillImageNotification object:self];
        self.stillImage = nil;
}];

There is no other image processing or manipulation.

My app works with the code above. I'm just trying to understand why the orientation constants must be reversed for landscape orientations. Thanks!

Answer 1

Heh, it seems nobody felt like chiming in on this one. Turns out the answer is straightforward. Images captured via the stillImageOutput captureStillImageAsynchronouslyFromConnection:... method always end up with the following properties:

• UIImage orientation = always UIImageOrientationRight regardless of device orientation
• UIImage size = W x H (e.g. portrait width x portrait height, depends on your camera resolution)
• CGImage size = depends on device orientation (e.g. portrait or landscape)

So the solution to rotate the image up is to use the device orientation in conjunction with the CGImage size to apply an appropriate affine transform. As I'm answering my own question, I'm not the solution in code but I ended up writing a routine called:

- (UIImage *)imageRotatedUpForDeviceOrientation:(UIDeviceOrientation)deviceOrientation

in a UIImage category containing various image processing enhancements.

EDIT - Implementation Example

I've received a number of requests for functional code on this. I've extracted the relevant implementation from a working app.

// this method is implemented in your capture session manager (wherever AVCaptureSession is used)
// capture a still image and save the device orientation
- (void)captureStillImage
{
    UIDeviceOrientation currentDeviceOrientation = UIDevice.currentDevice.orientation;
      [self.stillImageOutput
      captureStillImageAsynchronouslyFromConnection:self.videoConnection
      completionHandler:^(CMSampleBufferRef imageSampleBuffer, NSError *error) {
          NSData *imageData = [AVCaptureStillImageOutput jpegStillImageNSDataRepresentation:imageSampleBuffer];
          if (imageData) {
              UIImage *image = [UIImage imageWithData:imageData];
              NSDictionary *captureInfo = {
                  @"image" : image,
                  @"deviceOrientation" : @(currentDeviceOrientation)
              };
              // TODO: send image & orientation to delegate or post notification to observers
          }
          else {
              // TODO: handle image capture error
          }
    }];
}

// this method rotates the UIImage captured by the capture session manager based on the
// device orientation when the image was captured
- (UIImage *)imageRotatedUpFromCaptureInfo:(NSDictionary *)captureInfo
{
    UIImage *image = [captureInfo objectForKey:@"image"];
    UIDeviceOrientation deviceOrientation = [[captureInfo objectForKey:@"deviceOrientation"] integerValue];
    UIImageOrientation rotationOrientation = [self rotationNeededForImageCapturedWithDeviceOrientation:deviceOrientation];
    // TODO: scale the image if desired
    CGSize newSize = image.size;
    return [imageScaledToSize:newSize andRotatedByOrientation:rotationOrientation];
}

// return a scaled and rotated an image
- (UIImage *)imageScaledToSize:(CGSize)newSize andRotatedByOrientation:(UIImageOrientation)orientation
{
    CGImageRef imageRef = self.CGImage;    
    CGRect imageRect = CGRectMake(0.0, 0.0, newSize.width, newSize.height);
    CGRect contextRect = imageRect;
    CGAffineTransform transform = CGAffineTransformIdentity;

    switch (orientation)
    {
        case UIImageOrientationDown: { // rotate 180 deg
            transform = CGAffineTransformTranslate(transform, imageRect.size.width, imageRect.size.height);
            transform = CGAffineTransformRotate(transform, M_PI);
        } break;

        case UIImageOrientationLeft: { // rotate 90 deg left
            contextRect = CGRectTranspose(contextRect);
            transform = CGAffineTransformTranslate(transform, imageRect.size.height, 0.0);
            transform = CGAffineTransformRotate(transform, M_PI / 2.0);
        } break;

        case UIImageOrientationRight: { // rotate 90 deg right
            contextRect = CGRectTranspose(contextRect);
            transform = CGAffineTransformTranslate(transform, 0.0, imageRect.size.width);
            transform = CGAffineTransformRotate(transform, 3.0 * M_PI / 2.0);
        } break;

        case UIImageOrientationUp: // no rotation
        default:
            break;
    }

    CGBitmapInfo bitmapInfo = CGImageGetBitmapInfo(imageRef);
    CGColorSpaceRef colorSpaceRef = CGImageGetColorSpace(imageRef);

    // madify bitmapInfo to work with PNG if necessary
    if (bitmapInfo == kCGImageAlphaNone) {
        bitmapInfo = kCGImageAlphaNoneSkipLast;
    }
    else if (bitmapInfo == kCGImageAlphaLast) {
        bitmapInfo = kCGImageAlphaPremultipliedLast;
    }

    // Build a context that's the same dimensions as the new size
    CGContextRef context = CGBitmapContextCreate(NULL,
                                                 contextRect.size.width,
                                                 contextRect.size.height,
                                                 CGImageGetBitsPerComponent(imageRef),
                                                 0,
                                                 colorSpaceRef,
                                                 bitmapInfo);


    CGContextConcatCTM(context, transform);
    CGContextDrawImage(context, imageRect, imageRef);

    // Get the rotated image from the context and a UIImage
    CGImageRef rotatedImageRef = CGBitmapContextCreateImage(context);
    UIImage *rotatedImage = [UIImage imageWithCGImage:rotatedImageRef];

    // Clean up
    CGImageRelease(rotatedImageRef);
    CGContextRelease(context);

    return rotatedImage;
}

// return the UIImageOrientation needed for an image captured with a specific deviceOrientation
- (UIImageOrientation)rotationNeededForImageCapturedWithDeviceOrientation:(UIDeviceOrientation)deviceOrientation
{
    UIImageOrientation rotationOrientation;
    switch (deviceOrientation) {
        case UIDeviceOrientationPortraitUpsideDown: {
            rotationOrientation = UIImageOrientationLeft;
        } break;

        case UIDeviceOrientationLandscapeRight: {
            rotationOrientation = UIImageOrientationDown;
        } break;

        case UIDeviceOrientationLandscapeLeft: {
            rotationOrientation = UIImageOrientationUp;
        } break;

        case UIDeviceOrientationPortrait:
        default: {
            rotationOrientation = UIImageOrientationRight;
        } break;
    }
    return rotationOrientation;
}

Answer 2

As for why you need AVCaptureVideoOrientationLandscapeRight when the device's orientation is UIDeviceOrientationLandscapeLeft, that's because, for some reason, UIDeviceOrientationLandscapeLeft == UIInterfaceOrientationLandscapeRight, and the AVCaptureVideoOrientations are following the UIInterfaceOrientation convention.

Also, UIDeviceOrientation incudes other options like UIDeviceOrientationFaceUp, UIDeviceOrientationFaceDown, and UIDeviceOrientationUnknown. If you're having your interface rotate to match the device's orientation, you could try getting the UIDeviceOrientation from [UIApplication sharedApplication].statusBarOrientation instead.

Answer 3

Actually, there is no mistake in your implementation and you can eliminate the entire switch case.

Why switch case can be eliminated:

The corressponding values of device and video orientations are numerically equal, i.e. as long as device orientation is not unknown, face up or face down, you can equate UIDeviceOrientation and AVCaptureVideoOrientation

About the confusing nomenclature:

Device orientation LandscapeLeft => the top of your phone is on the left.

So, just imagine where the starting point of video is..... that's right, TOP RIGHT corner => Landscape right.

Similarly in the other landscape orientation, the video orientation is away from the top of the phone. This isn't confusing in case of portrait and upside down

Proof

When you capture a still Image, check the orientation EXIF value. For landscape left, the EXIF orientation is 3 (180 degrees inversion) For landscape right, the EXIF orientation is 1 (no rotation)

When you are displaying, your exif orientation is converted to corresponding UIImageOrientation, so there should be no inversion.

In the case of front camera, I have noticed that EXIF values, simply follow the video orientation values (mirror effect is not taken care of). Only 1, 3, 6, 8 values of EXIF are used.

Also, the mirror is always along the line from top of device to bottom So, landscape will seem upside down while portrait and upside down portrait is just mirrored