How to create UIImage from AVCapturePhoto with the correct orientation?

Question

I call the AVFoundation delegate AVFoundation to process the photo, but I find it difficult to convert the generated AVCapturePhoto to UIImage with the correct orientation. Although the procedure below is successful, I always focus on the correctness of UIImage ( UIImage.imageOrientation = 3). I have no way to provide orientation when using UIImage(data: image) , and trying the first time using photo.cgImageRepresentation()?.takeRetainedValue() also does not help. Please, help.

The orientation of the image here is crucial, as the resulting image is transferred to the Vision Framework workflow.

 func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) { // capture image finished print("Image captured.") if let imageData = photo.fileDataRepresentation() { if let uiImage = UIImage(data: imageData){ // do stuff to UIImage } } } 

UPDATE 1: After reading the Apple Photo Capture Programming Guide (deprecated for iOS11), I was able to discover one thing that I did wrong:

1. Each time you call capture ( self.capturePhotoOutput.capturePhoto ), you must establish a connection with the PhotoOutput object and update its orientation so that it matches the orientation of the device at the time of shooting. To do this, I created the UIDeviceOrientation extension and used it in the snapPhoto() function that I created to invoke the capture procedure and wait for the didFinishProcessingPhoto delegate method to didFinishProcessingPhoto . I added a snapshot of the code because here, it seems, the delimiters of the sample code do not display them correctly.

Update 2 Link to the full project on GitHub: https://github.com/agu3rra/Out-Loud

Answer 1

Final update: I conducted several experiments with the application and came to the following conclusions:

1. kCGImagePropertyOrientation does not seem to affect the orientation of the captured image inside your application, and it only changes depending on the orientation of the device if you update the photoOutput connection every time you intend to call the capturePhoto method. So:

    func snapPhoto() { // prepare and initiate image capture routine // if I leave the next 4 lines commented, the intented orientation of the image on display will be 6 (right top) - kCGImagePropertyOrientation let deviceOrientation = UIDevice.current.orientation // retrieve current orientation from the device guard let photoOutputConnection = capturePhotoOutput.connection(with: AVMediaType.video) else {fatalError("Unable to establish input>output connection")}// setup a connection that manages input > output guard let videoOrientation = deviceOrientation.getAVCaptureVideoOrientationFromDevice() else {return} photoOutputConnection.videoOrientation = videoOrientation // update photo output connection to match device orientation let photoSettings = AVCapturePhotoSettings() photoSettings.isAutoStillImageStabilizationEnabled = true photoSettings.isHighResolutionPhotoEnabled = true photoSettings.flashMode = .auto self.capturePhotoOutput.capturePhoto(with: photoSettings, delegate: self) // trigger image capture. It appears to work only if the capture session is running. } 

2. Viewing the generated images in the debugger showed me how they are generated, so I could output the required rotation ( UIImageOrientation ) so that it would be displayed in a vertical position. In other words: the UIImageOrientation update talks about how the image should rotate so you can see it in the correct orientation. So, I came to the following table:

3. I had to upgrade the UIDeviceOrientation extension to a rather unintuitive form:

    extension UIDeviceOrientation { func getUIImageOrientationFromDevice() -> UIImageOrientation { // return CGImagePropertyOrientation based on Device Orientation // This extented function has been determined based on experimentation with how an UIImage gets displayed. switch self { case UIDeviceOrientation.portrait, .faceUp: return UIImageOrientation.right case UIDeviceOrientation.portraitUpsideDown, .faceDown: return UIImageOrientation.left case UIDeviceOrientation.landscapeLeft: return UIImageOrientation.up // this is the base orientation case UIDeviceOrientation.landscapeRight: return UIImageOrientation.down case UIDeviceOrientation.unknown: return UIImageOrientation.up } } } 

4. This is what my last delegate method looks like. Displays the image in the expected orientation.

    func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) { // capture image finished print("Image captured.") let photoMetadata = photo.metadata // Returns corresponting NSCFNumber. It seems to specify the origin of the image // print("Metadata orientation: ",photoMetadata["Orientation"]) // Returns corresponting NSCFNumber. It seems to specify the origin of the image print("Metadata orientation with key: ",photoMetadata[String(kCGImagePropertyOrientation)] as Any) guard let imageData = photo.fileDataRepresentation() else { print("Error while generating image from photo capture data."); self.lastPhoto = nil; self.controller.goToProcessing(); return } guard let uiImage = UIImage(data: imageData) else { print("Unable to generate UIImage from image data."); self.lastPhoto = nil; self.controller.goToProcessing(); return } // generate a corresponding CGImage guard let cgImage = uiImage.cgImage else { print("Error generating CGImage");self.lastPhoto=nil;return } guard let deviceOrientationOnCapture = self.deviceOrientationOnCapture else { print("Error retrieving orientation on capture");self.lastPhoto=nil; return } self.lastPhoto = UIImage(cgImage: cgImage, scale: 1.0, orientation: deviceOrientationOnCapture.getUIImageOrientationFromDevice()) print(self.lastPhoto) print("UIImage generated. Orientation:(self.lastPhoto.imageOrientation.rawValue)") self.controller.goToProcessing() } func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) { print("Just about to take a photo.") // get device orientation on capture self.deviceOrientationOnCapture = UIDevice.current.orientation print("Device orientation: \(self.deviceOrientationOnCapture.rawValue)") } 

Answer 2

I was successful doing this:

 func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) { let cgImage = photo.cgImageRepresentation()!.takeRetainedValue() let orientation = photo.metadata[kCGImagePropertyOrientation as String] as! NSNumber let uiOrientation = UIImage.Orientation(rawValue: orientation.intValue)! let image = UIImage(cgImage: cgImage, scale: 1, orientation: uiOrientation) } 

This is based on what Apple mentions in its docs:

Each time you access this method, AVCapturePhoto generates a new CGImageRef. Supported by a compressed container (such as HEIC), CGImageRepresentation is decoded lazily as needed. Supported by an uncompressed format such as BGRA, it is copied to a separate backup buffer, whose lifetime is not related to the lifetime of AVCapturePhoto. For a 12 megapixel image, BGRA CGImage represents ~ 48 megabytes per call. If you intend to use CGImage only for rendering on the screen, use previewCGImageRepresentation instead. Note that the physical rotation of the CGImageRef is the same as the rotation of the main image. Exif orientation has not been applied. If you want to apply rotation when working with UIImage, you can do this by querying the photo metadata value [kCGImagePropertyOrientation] and passing it as an orientation parameter to + [UIImage imageWithCGImage: scale: direction:]. RAW images always return a CGImageRepresentation nil. If you want to create a CGImageRef from a RAW image, use CIRAWFilter in the CoreImage platform.

Answer 3

Inside AVCapturePhoto Im pretty sure that you will find the metadata object of the so-called CGImageProperties .
Inside this word you will find the EXIF ​​dictionary for orientation. The next step is only to orient and create the image in accordance with this.
I have no experience using AVCapturePhotoOutput , but I have used the old way.
Note that the EXIF ​​dictionary is displayed differently in UIImageOrientation.
Here is an article an article I wrote many years ago, but the basic principle is still valid.
This question will point you to some implementations, it is also quite old, I'm sure that in the latest version they released a simpler API, but it will still guide you should solve the problem.

Answer 4

Updated extension provided by Andre that works with Swift 4.2:

 import Foundation import UIKit extension UIDeviceOrientation { var imageOrientation: UIImage.Orientation { switch self { case .portrait, .faceUp: return .right case .portraitUpsideDown, .faceDown: return .left case .landscapeLeft: return .up case .landscapeRight: return .down case .unknown: return .up } } } 

Answer 5

To create our image with the correct orientation, we need to enter the correct UIImage.Orientation when we initialize the image.

It is best to use the CGImagePropertyOrientation which is returned from the photoOutput delegate to get the exact orientation that the camera session was in when the picture was taken. The only problem here is that although the enumeration values ​​between UIImage.Orientation and CGImagePropertyOrientation same, the raw values ​​are not. Apple offers a simple mapping to fix this.

https://developer.apple.com/documentation/imageio/cgimagepropertyorientation

Here is my implementation:

AVCapturePhotoCaptureDelegate

 func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) { if let _ = error { // Handle Error } else if let cgImageRepresentation = photo.cgImageRepresentation(), let orientationInt = photo.metadata[String(kCGImagePropertyOrientation)] as? UInt32, let imageOrientation = UIImage.Orientation.orientation(fromCGOrientationRaw: orientationInt) { // Create image with proper orientation let cgImage = cgImageRepresentation.takeUnretainedValue() let image = UIImage(cgImage: cgImage, scale: 1, orientation: imageOrientation) } } 

Mapping Extension

 extension UIImage.Orientation { init(_ cgOrientation: CGImagePropertyOrientation) { // we need to map with enum values becuase raw values do not match switch cgOrientation { case .up: self = .up case .upMirrored: self = .upMirrored case .down: self = .down case .downMirrored: self = .downMirrored case .left: self = .left case .leftMirrored: self = .leftMirrored case .right: self = .right case .rightMirrored: self = .rightMirrored } } /// Returns a UIImage.Orientation based on the matching cgOrientation raw value static func orientation(fromCGOrientationRaw cgOrientationRaw: UInt32) -> UIImage.Orientation? { var orientation: UIImage.Orientation? if let cgOrientation = CGImagePropertyOrientation(rawValue: cgOrientationRaw) { orientation = UIImage.Orientation(cgOrientation) } else { orientation = nil // only hit if improper cgOrientation is passed } return orientation } }