Decode sound samples from hls stream on ios?

Question

I am trying to decode sound samples from a remote HLS stream (m3u8) on an iOS device for further data processing, for example. write to file. The reference stream is http://devimages.apple.com/iphone/samples/bipbop/bipbopall.m3u8 .

Using AVURLAsset in combination with AVPlayer, I can show the video as a preview on CALayer. I can also get raw video data (CVPixelBuffer) using AVPlayerItemVideoOutput. Sound is also heard through the speaker of the iOS device.

This is the code I'm currently using for AVURLAsset and AVPlayer:

NSURL* url = [NSURL URLWithString:@"http://devimages.apple.com/iphone/samples/bipbop/bipbopall.m3u8"]; AVURLAsset *asset = [AVURLAsset URLAssetWithURL:url options:nil]; NSString *tracksKey = @"tracks"; [asset loadValuesAsynchronouslyForKeys:@[tracksKey] completionHandler: ^{ dispatch_async(dispatch_get_main_queue(), ^{ NSError* error = nil; AVKeyValueStatus status = [asset statusOfValueForKey:tracksKey error:&error]; if (status == AVKeyValueStatusLoaded) { NSDictionary *settings = @ { (id)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_32BGRA), @"IOSurfaceOpenGLESTextureCompatibility": @YES, @"IOSurfaceOpenGLESFBOCompatibility": @YES, }; AVPlayerItemVideoOutput* output = [[AVPlayerItemVideoOutput alloc] initWithPixelBufferAttributes:settings]; AVPlayerItem* playerItem = [AVPlayerItem playerItemWithAsset:asset]; [playerItem addOutput:output]; AVPlayer* player = [AVPlayer playerWithPlayerItem:playerItem]; [player setVolume: 0.0]; // no preview audio self.playerItem = playerItem; self.player = player; self.playerItemVideoOutput = output; AVPlayerLayer* playerLayer = [AVPlayerLayer playerLayerWithPlayer: player]; [self.preview.layer addSublayer: playerLayer]; [playerLayer setFrame: self.preview.bounds]; [playerLayer setVideoGravity: AVLayerVideoGravityResizeAspectFill]; [self setPlayerLayer: playerLayer]; [[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(playerItemNewAccessLogEntry:) name:AVPlayerItemNewAccessLogEntryNotification object:self.playerItem]; [_player addObserver:self forKeyPath:@"status" options:NSKeyValueObservingOptionNew context:&PlayerStatusContext]; [_playerItem addObserver:self forKeyPath:@"status" options:NSKeyValueObservingOptionNew context:&PlayerItemStatusContext]; [_playerItem addObserver:self forKeyPath:@"tracks" options:0 context:nil]; } }); }]; -(void) observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary *)change context:(void *)context { if (self.player.status == AVPlayerStatusReadyToPlay && context == &PlayerStatusContext) { [self.player play]; } } 

To get the raw HLS stream video, I use:

 CVPixelBufferRef buffer = [self.playerItemVideoOutput copyPixelBufferForItemTime:self.playerItem.currentTime itemTimeForDisplay:nil]; if (!buffer) { return; } CMSampleBufferRef newSampleBuffer = NULL; CMSampleTimingInfo timingInfo = kCMTimingInfoInvalid; timingInfo.duration = CMTimeMake(33, 1000); int64_t ts = timestamp * 1000.0; timingInfo.decodeTimeStamp = CMTimeMake(ts, 1000); timingInfo.presentationTimeStamp = timingInfo.decodeTimeStamp; CMVideoFormatDescriptionRef videoInfo = NULL; CMVideoFormatDescriptionCreateForImageBuffer( NULL, buffer, &videoInfo); CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault, buffer, true, NULL, NULL, videoInfo, &timingInfo, &newSampleBuffer); // do something here with sample buffer... CFRelease(buffer); CFRelease(newSampleBuffer); 

Now I would like to access the raw audio data, but so far no luck. I tried using MTAudioProcessingTap as described here:
http://venodesigns.net/2014/01/08/recording-live-audio-streams-on-ios/
Unfortunately, I could not get this to work correctly. I managed to access the base active type of the AVPlayerItem object, but the preparation and process callbacks of the MTAudioProcessingTap are never called. I am not sure that I am on the right track here.

AVPlayer reproduces the sound of the stream through the speaker, so internally the sound seems to be available as the original audio data. Can I access unprocessed audio data? If this is not possible with AVPlayer, are there other approaches?

If possible, I would not want to use ffmpeg, because the hardware decoder of the iOS device should be used to decode the stream.