You can use the SetBuffer() overload where you do not have to pass a byte[] buffer - just adjust the existing index and count to be used - then after the operation set count back to it's original value:
mySocketAsyncEventArgs.SetBuffer(mySocketAsyncEventArgs.Offset, numOfBytesWritten);
mySocket.SendToAsync(mySocketAsyncEventArgs);
...
mySocketAsyncEventArgs.SetBuffer(mySocketAsyncEventArgs.Offset, originalBufferSize);