multicast and TCP are mutually exclusive.
Implementing reliable UDP delivery is nuts. No one has been doing this since the 1980s, and it's impossible to do as well as any cheap TCP stack, in terms of performance and BW overhead. Correction: sometimes this is done manually, but only by exotic vehicles, such as very long or narrow pipes.
N ^ 2 connections are not very stupid. Connection with 1 Hz keepalives is not expensive. What are the costs of traffic. This is what you need to concentrate on.
source share