First of all, you checked if you can make a decision from this application? This is not obvious from the site if they managed to ensure that their virtual serial ports are fully integrated into the system.
If there is a way to do this from user space, I don't know about that. The IOKit API for user space usually does not allow class instances, not to mention the new device driver classes. Perhaps you can somehow convince Cocoa libraries to find it even though it is not registered in the kernel.
I don’t know if you can get away with creating the "dummy" serial port in the kernel, and then move your tty to its place in / dev from your user space demonstrator. Perhaps this is an option.
If you need to do all this in the kernel:
The virtual driver itself should not work too much, although it will take some time to speed up work with the dev kernel. Unfortunately, the documentation is pretty thin for serial port drivers - the key subclasses the abstract class IOSerialDriverSync . Almost the only description I've seen is Ole Henry Halvorsen OSX and iOS Kernel. It also has an example snippet for read and write operations. ( disclosure: I was one of the reviewer technicians for this book, I get no incentive to recommend - in this case, this is literally the only documentation I know of). You can find the source for the full serial port driver in the Apple USBCDC driver , AppleUSBCDCDMM is the class that actually represents the node serial port.
It is relatively simple to open the so-called “control core” in the kernel, the individual APIs described here ; from user space, you use the usual send / recv BSD APIs. (this is also described in the aforementioned book). Then your daemon can connect to this, and all you have to do is push the data between the socket and the virtual serial port device. You will need to handle disconnect events and, of course, correctly.
However, I think this is possible as the first kernel project for an experienced C programmer (with some C ++).
I hope this helps!