How to handle real-time media analysis on a web server?

Question

How to handle real-time media analysis on a web server?

Usage example . I am developing a web application to help students learn to read. A student writes while reading text in a web application. The signal is sent by the 200 ms segment to the backend and analyzed before the student finishes reading in order to give live feedback while reading. The server will send feedback after each analysis segment.

In a web application, the code is as follows:

navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(stream => {
    const mediaRecorder = new MediaRecorder(stream)
    mediaRecorder.start(200)

    mediaRecorder.ondataavailable = event => {
        socket.emit('my_event', Blob([event.data]))
    }
})

On chrome, a media type is created by webm. I am wondering how to process data on a server so that I can analyze media from numpy to the end of the recording.

At the moment, I could not find a better way than something like:

from pydub import AudioSegment
def blobToSignal(blob, is_first_sequence):
    webm_header = b'\x1aE...'
    fp = tempfile.NamedTemporaryFile()
    fp.write(blob) if is_first_sequence else fp.write(webm_header + blob)
    fp.seek(0)
    samples = AudioSegment.from_file(fp.name, 'webm').get_array_of_samples()
    fp.close()
    return samples # this is almost a numpy array (analyzable)

I tried changing the front to return a Float32Array instead of webm:

navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(stream => {
    audio_context = new AudioContext()
    var audioInput = audio_context.createMediaStreamSource(stream)
    var recorder = audio_context.createScriptProcessor(8096, 1, 1)
    recorder.onaudioprocess = event => {
        socket.emit(
            'my_event',
            Array.from(event.inputBuffer.getChannelData(0))
        )
    }
    audioInput.connect(recorder)
    recorder.connect(audio_context.destination)