You can mix apples and oranges in your question. DirectSound provides an API for accessing sound cards; H.323 and SIP are call standards.
Silverlight 4 initially provides several simplified APIs that are roughly equivalent to DirectSound, so you can access your microphone, sound card, and webcams. See http://msdn.microsoft.com/en-us/library/ff602282(VS.95).aspx for more details.
But once you get the sound, you still need to encode, transmit, mix, distribute, and then receive it. Where it gets complicated. Flash supports all this natively; Silverlight does not. The best open source implementation right now for this has been compiled by SocketCoder and available on CodePlex , but uses a proprietary signaling / streaming protocol, a proprietary and very inefficient video codec, and its own media server.
From what I can say, the best option to accomplish what you want in Silverlight is the closed source solution provided by StreamCoders . It's not open-source, but it does support a reasonable set of open standards, which means that you can probably use a third-party open source media server. You would like to test to make sure that this is what you need, of course.
Alternatively, you can try your hand at porting any of the various open source solutions to C #. I did this with the Speex pre-processor: it is complicated, but not as complicated as you think. You could probably do something similar with openH323 and the various C codecs that are there. You might even try changing the version of Silverlight RTMP with FluorineFx (http://www.fluorinefx.com/) to do what you need. Of course, none of this can be simple.