VNTrackRectangleRequest internal error

Question

I am trying to get a simple rectangle tracking controller, and I can detect that the detection of the rectangle is fine, but the tracking request always fails for the reason I cannot find.

Sometimes a tracking request triggers its callback several times before a failure, and in other cases it is interrupted immediately before a single callback occurs. I feel this is due to the way I send requests, but I can’t understand it.

Here is the code for the view controller

class TestController: UIViewController, AVCaptureVideoDataOutputSampleBufferDelegate { // Video capture private var videoSession = AVCaptureSession() private var videoLayer: AVCaptureVideoPreviewLayer! // Detection private var detectionRequest: VNDetectRectanglesRequest? private let sequenceHandler = VNSequenceRequestHandler() // Tracking private var trackingRequest: VNTrackRectangleRequest? private var shape: Detection? private var pixelBuffer: CVImageBuffer? // MARK: Setup override func viewWillAppear(_ animated: Bool) { super.viewWillAppear(animated) startVideoFeed() } override func viewDidLayoutSubviews() { videoLayer.frame = view.layer.bounds } override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) { startDetectingRectangles() } private func startDetectingRectangles() { let request = VNDetectRectanglesRequest(completionHandler: didDetectRectangle) request.maximumObservations = 1 request.minimumSize = 0.07 request.minimumConfidence = 0.9 request.minimumAspectRatio = 0.5 request.maximumAspectRatio = 2 request.quadratureTolerance = 10 detectionRequest = request } private func didDetectRectangle(request: VNRequest, error: Error?) { // Fetch results of the correct type guard let observations = request.results, observations.count > 0 else { return } let results = observations.map { $0 as? VNRectangleObservation } for case let rectangle? in results { detectionRequest = nil let request = VNTrackRectangleRequest(rectangleObservation: rectangle, completionHandler: didTrackRectangle) trackingRequest = request } } private func didTrackRectangle(request: VNRequest, error: Error?) { // Fetch results of the correct type guard let observation = request.results?.first as? VNRectangleObservation else { return } // Create or update UI } // Start capturing video frames private func startVideoFeed() { // Session config videoSession.sessionPreset = .photo // Create device and input to device guard let captureDevice = AVCaptureDevice.default(for: .video), let deviceInput = try? AVCaptureDeviceInput(device: captureDevice) else { fatalError("Error setting up capture device.") } // Setup device output let deviceOutput = AVCaptureVideoDataOutput() deviceOutput.setSampleBufferDelegate(self, queue: DispatchQueue.global(qos: .default)) // Set input and output videoSession.addInput(deviceInput) videoSession.addOutput(deviceOutput) // Setup video display layer videoLayer = AVCaptureVideoPreviewLayer(session: videoSession) videoLayer.frame = view.bounds videoLayer.videoGravity = .resizeAspectFill view.layer.addSublayer(videoLayer) videoSession.startRunning() } func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) guard let pixelBuffer = pixelBuffer else { return } do { var requests: [VNRequest] = [] if let detectionRequest = detectionRequest { requests.append(detectionRequest) } if let trackingRequest = trackingRequest { requests.append(trackingRequest) } try sequenceHandler.perform(requests, on: pixelBuffer, orientation: .right) } catch { print(error) } } } 

Print Error:

Domain error = com.apple.vis Code = 9 "Internal error: tracking one of the corners failed, trust = 0.000000; threshold = 0.650000" UserInfo = {NSLocalizedDescription = Internal error: tracking one of the corners failed, trust = 0.000000; threshold = 0.650000}

This is displayed every frame after the first tracking request, and the request is never restored. This is a try sequenceHandler.perform(requests, on: pixelBuffer, orientation: .right) line try sequenceHandler.perform(requests, on: pixelBuffer, orientation: .right) that throws an error (which I then print).

I looked at examples of observing objects posted around the Internet, and they seem to require that each track be requested for each object to track the object. I tried to create a new VNTrackRectangleRequest in the didTrackRectangle function didTrackRectangle that each frame has a new request, but I get the same problem.

Any help on this is really appreciated.

Answer 1

I was able to run your code without any problems, at least when tracking a clearly visible rectangle.

You should probably create a new VNSequenceRequestHandler when the tracking request fails. It seems that when the tracking request loses track of the object for too long, it will not be able to recover and start tracking it again. It also means that they will never be released and will continue to produce errors. If you continue to add new tracking requests to your request handler, you will quickly exceed the limit for concurrent tracking requests.

If you can foresee the end of the tracking sequence, you can set the isLastFrame property of your tracking request to true , and it will be passed to the pool of available tracking tools when the current frame completes processing.

Answer 2

I saw this error every frame, and to fix it, I found that I needed to call performRequests:onCVPixelBuffer:orientation:error: with the correct orientation .

(For portrait orientation with a rear view camera, I believe that the orientation is kCGImagePropertyOrientationRight ).

From the documentation :

For each such request, call the request handlers of the sequence executeRequests: onCVPixelBuffer: orientation: error: method, making sure to pass the orientation of the video readers to ensure proper tracking.