Yes, exactly what you described is possible in Firefox by using the WebAudio API. At the current time it is not possible in Chrome because it cannot create a MediaStreamAudioSourceNode from WebRTC streams (I hope this limitation will go away soon). Thus, the moderator's browser must be Firefox. The other peers can use other browsers.
This way you can set up a conference call with 10 peers, all of them only connecting to the moderator, thus using only 10 WebRTC connections.
What you have forgotten to mention is that you also have to mix in the moderator's audio for each peer.
With the WebAudio API you also could do some fancy things like per-peer audio visualization, muting, volume control, mixing in of audio files, etc.