I implemented something like this because I have a visceral dislike of Java's EventListenerList. First, you implement a common receiver. I determined the listener based on the event that it was receiving, mainly using one method
interface GenericListener<T extends Event> { public void handle(T t); }
This will help you define ListenerA, ListernerB, etc. Although you can do this with ListenerA, ListenerB, etc., everyone extends some databases, such as MyListener. Both methods have pros and cons.
Then I used CopyOnWriteArraySet
to hold all of these listeners. Dialing is something to keep in mind, because all too often, listeners get twice as slow encoders. YMMV. But, effectively, you have a Collection<GenericListener<T extends Event>> or a Collection<MyListener>
Now, as you have discovered, with type erasure, a collection can contain only one type of listener. This is often a problem. Decision. Use a card.
Since I base everything on an event, I used
Map<Class<T extends Event>, Collection<GenericListener<T extends Event>>>
depending on the class of the event, get a list of listeners who want to receive this event.
Your alternative is to base it on a listener class
Map<Class<T extends MyListener>, Collection<MyListener>>
Probably some typos above ...