Try resetting your memory stream position to zero before you copy it to response:
// Close the document.
spreadsheetDocument.Close();
mem.Position = 0;
mem.CopyTo(response.OutputStream);
Alternatively to copying memory stream to response, you can return FileStreamResult
from your MVC Action:
public ActionResult XlsDocument()
{
// generating your document here...
// very important to reset memory stream to a starting position, otherwise you would get 0 bytes returned
memoryStream.Position = 0;
var resultStream = new FileStreamResult(memoryStream, "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";);
resultStream.FileDownloadName = String.Format("{0}.{1}", fileName, extension);
return resultStream;
}