Real-time midi input and audio synchronization

Question

I built a standalone version of the application, which so far has only been VST / audiounit. I provide audio support through rtaudio .

I would like to add MIDI support using rtmidi , but I don’t understand how to synchronize audio and MIDI parts.

In VST / audiounit land, I use MIDI events that have a timestamp indicating their offset in samples from the beginning of the sound block.

rtmidi provides delta time in seconds from the previous event, but I'm not sure how I should capture these events and how I can work out their time relative to the current sample in the audio stream.

How do plugin hosts do it?

I can understand how events can be accurate when played back, but it's not clear how they can be accurate when used in real time.

rtaudio gives me a callback function. I will work with a low block size (32 samples). I assume that I will pass a pointer to an rtmidi instance as part of the userdata callback, and then will call midiin-> getMessage (& message); inside an audio callback, but I'm not sure if this is clear to threads.

Thanks so much for the advice you can give me.

Answer 1

In your case, you do not need to worry about it. Your program should send MIDI events to the plugin with a time stamp of zero as soon as they arrive. I think you may have misunderstood the idea of ​​what it means to be an “exact model.”

As @Brad noted in his comment on your question, MIDI is really very slow. But this is only part of the problem ... when you work in a block environment, incoming MIDI events cannot be processed by the plugin before the block starts. When computers were slower and 512 block sizes (or god-forbid,> 1024) were normal, this introduced a non-trivial amount of latency, resulting in the device not sounding like a "tight" one. Therefore, sequencers have come up with a smart way to get around this problem. Since MIDI events are already known in advance, these events can be sent to the instrument one block earlier with an offset in the sample frames. Then the plugin receives these events at the beginning of the block and knows that it will not actually start processing them until the samples N pass. This is what “exact pattern” means in sequencers.

However, if you are dealing with a live input from a keyboard or some other MIDI device, it is impossible to "paint" these events. In fact, by the time you get them, the clock is already ticking! Therefore, these events should simply be sent to the plug-in at the beginning of the very next block with offset 0. Sequencers such as Ableton Live, which allow the plug-in to simultaneously receive both pre-ordered and live events, simply send any live events with an offset of 0 frames.

Since you are using a very small block size, the worst case scenario is 0.7 ms latency, which is not so bad. In the case of rtmidi, the timestamp is not the offset you need to plan, but the time at which the event was recorded. But since you only intend to receive live events (you are not writing a sequencer, are you?), You can simply transfer any incoming MIDI to the plug-in immediately.