We recently added two new methods on AudioBuffer, from which use copyFromChannel could be useful to you. It does what it says on the tin (put an ArrayBuffer in a channel of an AudioBuffer), maybe saving a copy in the process (the reasons and condition to avoid the copy are explained here).
I believe it's only implemented in Firefox for the moment, but I haven't checked.