Record audio and transfer data to UIWebView (JavascriptCore) on iOS 8/9

Question

We have an application, which is basically a UIWebView for a web application with great javascript support. The requirement that we are faced with is the ability to play the sound to the user, and then record the user, play this record for confirmation, and then send the audio to the server. This works on Chrome, Android and other platforms because this feature is built into the browser. No required code needed.

Unfortunately, the iOS webpage (iOS 8/9) does not have the ability to record sound.

The first workaround we tried was recording audio using AudioQueue and transferring data (LinearPCM 16bit) to JS AudioNode so that the web application could process iOS sound exactly like other platforms. This went so far as to allow us to transfer audio to JS, but the application will eventually fail with a memory access error or the javascript side simply cannot keep up with the data being sent.

The next idea was to save the audio recording to a file and send partial JS audio data for visual feedback - the main audio visualizer displayed only during recording.

The audio recording is recorded and played back in the WAVE file, as Linear PCM has a 16-bit format. We are stuck in the JS visualizer. The unsigned linear PCM is expected to be 8 bits, so I added a conversion step that might be wrong. I tried several different methods, mainly found on the Internet, and did not find what works, which makes me think that something else didn’t or didn’t work out before we even move on to the conversion step.

Since I don’t know what and where exactly the problem is, I will give the code below for sound recording and playback classes. Any suggestions can be resolved to solve this problem or somehow.

One of my ideas was to record in a different format (CAF) using different format flags. Considering the values ​​that are produced, unsigned 16-bit ints approach the maximum value. I rarely see something above +/- 1000. Is it because of the kLinearPCMFormatFlagIsPacked flag in AudioStreamPacketDescription? Removing this flag causes the audio file to not be created due to an invalid format. Maybe switching to CAF will work, but we need to convert to WAVE before sending the sound back to our server.

Or maybe my conversion from signed 16-bit to unsigned 8bit is wrong? I also tried bithitting and casting. The only difference is that with this conversion all sound values ​​are compressed to 125 and 130. The bit offset and casting are changed to 0-5 and 250-255. This does not really solve any problems on the part of JS.

The next step is, instead of transferring data to JS, run it through the FFT function and create values ​​that will be used directly by JS for the audio visualizer. I would rather find out if I did something clearly wrong before going in that direction.

AQRecorder.h - EDIT: Updated Audio Format for LinearPCM 32bit Float.

#ifndef AQRecorder_h #define AQRecorder_h #import <AudioToolbox/AudioToolbox.h> #define NUM_BUFFERS 3 #define AUDIO_DATA_TYPE_FORMAT float #define JS_AUDIO_DATA_SIZE 32 @interface AQRecorder : NSObject { AudioStreamBasicDescription mDataFormat; AudioQueueRef mQueue; AudioQueueBufferRef mBuffers[ NUM_BUFFERS ]; AudioFileID mAudioFile; UInt32 bufferByteSize; SInt64 mCurrentPacket; bool mIsRunning; } - (void)setupAudioFormat; - (void)startRecording; - (void)stopRecording; - (void)processSamplesForJS:(UInt32)audioDataBytesCapacity audioData:(void *)audioData; - (Boolean)isRunning; @end #endif 

AQRecorder.m - EDIT: Updated Audio Format for LinearPCM 32bit Float. Added FFT step to processSamplesForJS instead of sending direct audio data.

 #import <AVFoundation/AVFoundation.h> #import "AQRecorder.h" #import "JSMonitor.h" @implementation AQRecorder void AudioQueueCallback(void * inUserData, AudioQueueRef inAQ, AudioQueueBufferRef inBuffer, const AudioTimeStamp * inStartTime, UInt32 inNumberPacketDescriptions, const AudioStreamPacketDescription* inPacketDescs) { AQRecorder *aqr = (__bridge AQRecorder *)inUserData; if ( [aqr isRunning] ) { if ( inNumberPacketDescriptions > 0 ) { AudioFileWritePackets(aqr->mAudioFile, FALSE, inBuffer->mAudioDataByteSize, inPacketDescs, aqr->mCurrentPacket, &inNumberPacketDescriptions, inBuffer->mAudioData); aqr->mCurrentPacket += inNumberPacketDescriptions; [aqr processSamplesForJS:inBuffer->mAudioDataBytesCapacity audioData:inBuffer->mAudioData]; } AudioQueueEnqueueBuffer(inAQ, inBuffer, 0, NULL); } } - (void)debugDataFormat { NSLog(@"format=%i, sampleRate=%f, channels=%i, flags=%i, BPC=%i, BPF=%i", mDataFormat.mFormatID, mDataFormat.mSampleRate, (unsigned int)mDataFormat.mChannelsPerFrame, mDataFormat.mFormatFlags, mDataFormat.mBitsPerChannel, mDataFormat.mBytesPerFrame); } - (void)setupAudioFormat { memset(&mDataFormat, 0, sizeof(mDataFormat)); mDataFormat.mSampleRate = 44100.; mDataFormat.mChannelsPerFrame = 1; mDataFormat.mFormatID = kAudioFormatLinearPCM; mDataFormat.mFormatFlags = kLinearPCMFormatFlagIsFloat | kLinearPCMFormatFlagIsPacked; int sampleSize = sizeof(AUDIO_DATA_TYPE_FORMAT); mDataFormat.mBitsPerChannel = 32; mDataFormat.mBytesPerPacket = mDataFormat.mBytesPerFrame = (mDataFormat.mBitsPerChannel / 8) * mDataFormat.mChannelsPerFrame; mDataFormat.mFramesPerPacket = 1; mDataFormat.mReserved = 0; [self debugDataFormat]; } - (void)startRecording/ { [self setupAudioFormat]; mCurrentPacket = 0; NSString *recordFile = [NSTemporaryDirectory() stringByAppendingPathComponent: @"AudioFile.wav"]; CFURLRef url = CFURLCreateWithString(kCFAllocatorDefault, (CFStringRef)recordFile, NULL);; OSStatus *stat = AudioFileCreateWithURL(url, kAudioFileWAVEType, &mDataFormat, kAudioFileFlags_EraseFile, &mAudioFile); NSError *error = [NSError errorWithDomain:NSOSStatusErrorDomain code:stat userInfo:nil]; NSLog(@"AudioFileCreateWithURL OSStatus :: %@", error); CFRelease(url); bufferByteSize = 896 * mDataFormat.mBytesPerFrame; AudioQueueNewInput(&mDataFormat, AudioQueueCallback, (__bridge void *)(self), NULL, NULL, 0, &mQueue); for ( int i = 0; i < NUM_BUFFERS; i++ ) { AudioQueueAllocateBuffer(mQueue, bufferByteSize, &mBuffers[i]); AudioQueueEnqueueBuffer(mQueue, mBuffers[i], 0, NULL); } mIsRunning = true; AudioQueueStart(mQueue, NULL); } - (void)stopRecording { mIsRunning = false; AudioQueueStop(mQueue, false); AudioQueueDispose(mQueue, false); AudioFileClose(mAudioFile); } - (void)processSamplesForJS:(UInt32)audioDataBytesCapacity audioData:(void *)audioData { int sampleCount = audioDataBytesCapacity / sizeof(AUDIO_DATA_TYPE_FORMAT); AUDIO_DATA_TYPE_FORMAT *samples = (AUDIO_DATA_TYPE_FORMAT*)audioData; NSMutableArray *audioDataBuffer = [[NSMutableArray alloc] initWithCapacity:JS_AUDIO_DATA_SIZE]; // FFT stuff taken mostly from Apples aurioTouch example const Float32 kAdjust0DB = 1.5849e-13; int bufferFrames = sampleCount; int bufferlog2 = round(log2(bufferFrames)); float fftNormFactor = (1.0/(2*bufferFrames)); FFTSetup fftSetup = vDSP_create_fftsetup(bufferlog2, kFFTRadix2); Float32 *outReal = (Float32*) malloc((bufferFrames / 2)*sizeof(Float32)); Float32 *outImaginary = (Float32*) malloc((bufferFrames / 2)*sizeof(Float32)); COMPLEX_SPLIT mDspSplitComplex = { .realp = outReal, .imagp = outImaginary }; Float32 *outFFTData = (Float32*) malloc((bufferFrames / 2)*sizeof(Float32)); //Generate a split complex vector from the real data vDSP_ctoz((COMPLEX *)samples, 2, &mDspSplitComplex, 1, bufferFrames / 2); //Take the fft and scale appropriately vDSP_fft_zrip(fftSetup, &mDspSplitComplex, 1, bufferlog2, kFFTDirection_Forward); vDSP_vsmul(mDspSplitComplex.realp, 1, &fftNormFactor, mDspSplitComplex.realp, 1, bufferFrames / 2); vDSP_vsmul(mDspSplitComplex.imagp, 1, &fftNormFactor, mDspSplitComplex.imagp, 1, bufferFrames / 2); //Zero out the nyquist value mDspSplitComplex.imagp[0] = 0.0; //Convert the fft data to dB vDSP_zvmags(&mDspSplitComplex, 1, outFFTData, 1, bufferFrames / 2); //In order to avoid taking log10 of zero, an adjusting factor is added in to make the minimum value equal -128dB vDSP_vsadd(outFFTData, 1, &kAdjust0DB, outFFTData, 1, bufferFrames / 2); Float32 one = 1; vDSP_vdbcon(outFFTData, 1, &one, outFFTData, 1, bufferFrames / 2, 0); // Average out FFT dB values int grpSize = (bufferFrames / 2) / 32; int c = 1; Float32 avg = 0; int d = 1; for ( int i = 1; i < bufferFrames / 2; i++ ) { if ( outFFTData[ i ] != outFFTData[ i ] || outFFTData[ i ] == INFINITY ) { // NAN / INFINITE check c++; } else { avg += outFFTData[ i ]; d++; //NSLog(@"db = %f, avg = %f", outFFTData[ i ], avg); if ( ++c >= grpSize ) { uint8_t u = (uint8_t)((avg / d) + 128); //dB values seem to range from -128 to 0. NSLog(@"%i = %i (%f)", i, u, avg); [audioDataBuffer addObject:[NSNumber numberWithUnsignedInt:u]]; avg = 0; c = 0; d = 1; } } } [[JSMonitor shared] passAudioDataToJavascriptBridge:audioDataBuffer]; } - (Boolean)isRunning { return mIsRunning; } @end 

Play audio and record classes contrller Audio.h

 #ifndef Audio_h #define Audio_h #import <AVFoundation/AVFoundation.h> #import "AQRecorder.h" @interface Audio : NSObject <AVAudioPlayerDelegate> { AQRecorder* recorder; AVAudioPlayer* player; bool mIsSetup; bool mIsRecording; bool mIsPlaying; } - (void)setupAudio; - (void)startRecording; - (void)stopRecording; - (void)startPlaying; - (void)stopPlaying; - (Boolean)isRecording; - (Boolean)isPlaying; - (NSString *) getAudioDataBase64String; @end #endif 

Audio.m

 #import "Audio.h" #import <AudioToolbox/AudioToolbox.h> #import "JSMonitor.h" @implementation Audio - (void)setupAudio { NSLog(@"Audio->setupAudio"); AVAudioSession *session = [AVAudioSession sharedInstance]; NSError * error; [session setCategory:AVAudioSessionCategoryPlayAndRecord error:&error]; [session setActive:YES error:nil]; recorder = [[AQRecorder alloc] init]; mIsSetup = YES; } - (void)startRecording { NSLog(@"Audio->startRecording"); if ( !mIsSetup ) { [self setupAudio]; } if ( mIsRecording ) { return; } if ( [recorder isRunning] == NO ) { [recorder startRecording]; } mIsRecording = [recorder isRunning]; } - (void)stopRecording { NSLog(@"Audio->stopRecording"); [recorder stopRecording]; mIsRecording = [recorder isRunning]; [[JSMonitor shared] sendAudioInputStoppedEvent]; } - (void)startPlaying { if ( mIsPlaying ) { return; } mIsPlaying = YES; NSLog(@"Audio->startPlaying"); NSError* error = nil; NSString *recordFile = [NSTemporaryDirectory() stringByAppendingPathComponent: @"AudioFile.wav"]; player = [[AVAudioPlayer alloc] initWithContentsOfURL:[NSURL fileURLWithPath:recordFile] error:&error]; if ( error ) { NSLog(@"AVAudioPlayer failed :: %@", error); } player.delegate = self; [player play]; } - (void)stopPlaying { NSLog(@"Audio->stopPlaying"); [player stop]; mIsPlaying = NO; [[JSMonitor shared] sendAudioPlaybackCompleteEvent]; } - (NSString *) getAudioDataBase64String { NSString *recordFile = [NSTemporaryDirectory() stringByAppendingPathComponent: @"AudioFile.wav"]; NSError* error = nil; NSData *fileData = [NSData dataWithContentsOfFile:recordFile options: 0 error: &error]; if ( fileData == nil ) { NSLog(@"Failed to read file, error %@", error); return @"DATAENCODINGFAILED"; } else { return [fileData base64EncodedStringWithOptions:0]; } } - (Boolean)isRecording { return mIsRecording; } - (Boolean)isPlaying { return mIsPlaying; } - (void)audioPlayerDidFinishPlaying:(AVAudioPlayer *)player successfully:(BOOL)flag { NSLog(@"Audio->audioPlayerDidFinishPlaying: %i", flag); mIsPlaying = NO; [[JSMonitor shared] sendAudioPlaybackCompleteEvent]; } - (void)audioPlayerDecodeErrorDidOccur:(AVAudioPlayer *)player error:(NSError *)error { NSLog(@"Audio->audioPlayerDecodeErrorDidOccur: %@", error.localizedFailureReason); mIsPlaying = NO; [[JSMonitor shared] sendAudioPlaybackCompleteEvent]; } @end 

The JSMonitor class is the bridge between the UIWebView javascriptcore and native code. I do not turn it on because it does nothing for audio other than transferring data / calls between these classes and JSCore.

EDIT

The audio format has changed to LinearPCM Float 32bit. Instead of sending audio, it is sent via the FFT function, and the dB values ​​are averaged and sent instead.

Answer 1

Core Audio is a pain to work with. Fortunately, AVFoundation provides AVAudioRecorder for video recording, and also gives you access to average and maximum audio power, which you can send back to your JavaScript to update the visualizer of the interface. From the docs :

An instance of the AVAudioRecorder class, called a sound recorder, provides the ability to record sound into your application. Using an audio recorder you can:

• Recording until the user stops recording.
• Recording for a specific duration
• Pause and resume recording
• Get audio level input that can be used to provide level measurement

This stack overflow question contains an example of using AVAudioRecorder .