It could be because the underlying stream (in your case a MemoryStream) is not positioned at the beginning when you return it to the client.
Try this just before the return statement:
stream.Position = 0
Also, these lines of code:
writer.Flush();
stream.Flush();
Are not required because the stream is Memory based. You only need those for disk or network streams where there could be bytes that still require writing.