I assume you want to implement that algorithm on the server (on the client it's a whole lot of more difficult).
Nevertheless, using your example GStreamer pipeline, you'll most likely end up with timing issues (every queue object may delay the audio stream from which point it will be impossible to cancel out any audio).
To illustrate your requirements, I've drawn a (simplified) pipeline (incoming streams are decoded before src[A-D] and the outgoing ones are encoded after stream[A-D]:
The yellow boxes are time-critical, therefore I think the easiest way to do this would be writing your own gstreamer element that does the adding and subtracting work. Then place queue elements right before your element's input (and after your element's outputs; to work around network latency issues).
Another note: GStreamer may use many threads for one pipeline (especially if it's not linear like in your case) so if you get a lot of clients, you might hit the thread limit on your server (see this SO question, I hit that limit once using GStreamer (ok, I could've placed everything in one or two pipelines, but I didn't want read errors to cause all of my audio streams to stop) and a rather simple custom plugin helped me deal with that problem (each pipeline used two threads afterwards instead of eight)).