Receiving data in C# through a data Stream and reading data through boost:asio http server example

Question

So my problem is I have a file that I would like to post to my boost:asio http server. The client application is in C#. So I try to do the following based on lots of posts here and even from Microsoft's own site.

string fileToSendServer = "Contains lots of data that needs to be written to file";
byte[] byteArray = UTF8Encoding.UTF8.GetBytes(fileToSendServer);

// Create a request using a URL that can receive a post. 
request = WebRequest.Create(destination);

// Set the Method property of the request to POST.
request.Method = "POST";
// Set the ContentType property of the WebRequest.
request.ContentType = "application/octet-stream";

// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
// Get the request stream.
using (dataStream = request.GetRequestStream())
{
    // Write the data to the request stream.
    dataStream.Write(byteArray, 0, byteArray.Length);
    dataStream.Flush();
    // Close the Stream object.
    dataStream.Close();
}

The client gets all the way to GetRequestStream() and this seems to send the headers over to the waiting server. The server processes the headers and never receives bytes from the Write().

I have tried to use Fiddler2 to watch the packets but my IT has locked my machine down so that I can't run Fiddler.

When I step through the code for the client all lines of code are executed. And when I step through on the server, only the headers are found on the stream.

The data that needs to be sent is generated by the client and not really a file until it is supposed to be written to file on the server side. The server is similar to one of the boost::asio http example servers. So I don't know if it is the client or the server. I have been through the code for the parsing of the incoming input and the data is never received. So I looked at the connection and as far as I can tell there is no blocking on the incoming data.

EDIT: Okay based on the answers (including my own) I made some significant progress. Half the problem was that the c# wasn't reading everything, before sending. Now the C# Client is reading and sending the data. The Client reports that the send command sent all of the bytes. But the boost:asio http server 3 is still not receiving all of the data. The issue is that I am using the example http server and the request module doesn't seem to be capturing all of the data. It reads in one byte at a time and seems to die at 0x1f48 bytes of content.

template <typename InputIterator>
boost::tuple<boost::tribool, InputIterator> parse(request& req,
    InputIterator begin, InputIterator end)
{
    while (begin != end)
    {
        boost::tribool result = consume(req, *begin++);
        if (result || !result)
        {
            return boost::make_tuple(result, begin);
        }
    }
    boost::tribool result = boost::indeterminate;
    if(state_ == getting_content) {
        result = boost::indeterminate;
    }
    return boost::make_tuple(result, begin);
}

The begin == end when only 8008 bytes have been read from the request.

Answer 1

Apparently the response.GetResponseStream() returns a Stream that will not Read() the entire response. In order to solve this issue I had to build up the following on the response:

byte[] data = new byte[response.ContentLength];
int offset = 0;
while (offset < response.ContentLength)
{
    offset += dataStream.Read(data, offset, (int)(response.ContentLength - offset));
}
List<byte> content = new List<byte>(2 * (int)response.ContentLength);

Then, I was able to write out through the second stream, which finally did receive all the content that was sent. Each time I checked the file output before this change, the file contained exactly what was in content. So, it seems that the whole problem was based on the initial response not "Reading" all of the data that was sent.

EDIT:

Further, the data coming into the server was not complete. The boost:asio http example code had issues for capturing all the content data. I would receive only a portion of the incoming content or file data. It seemed to stop at 8008 bytes every time. I found that the parse function was incorrect. So I modified it like this:

        template <typename InputIterator>
        boost::tuple<boost::tribool, InputIterator> parse(request& req,
            InputIterator begin, InputIterator end)
        {
            //may have to handle multipart here.
            while (begin != end)
            {
                boost::tribool result = consume(req, *begin++);
                if (result || !result)
                {
                    return boost::make_tuple(result, begin);
                }
            }
            boost::tribool result = boost::indeterminate;
            if(state_ == getting_content) {
                if(req.content.length() < content_length_) {
                    result = boost::indeterminate;
                } else {
                    result = true;//boost::indeterminate;
                }
            }
            return boost::make_tuple(result, begin);
        }

The biggest part was to make the check for the content length and return indeterminate for this function. And then ensure that the connection had code to read_some more, which the example code does have.

Answer 2

(This answer is based on the new information you provided on the current accepted answer)

All Stream classes that implement Read are only required to read up to the number of bytes passed in to count. That is the entire reason it returns a int.

Also because Read does not take a long for its parameters you still will miss data if the returned content is larger than Int32.MaxLength.

The correct way to do it is read in to a smaller buffer and let that buffer copy in to your larger array.

byte[] data = new byte[response.ContentLength];
byte[] buffer = new byte[4 << 10]; //4k buffer, you can tweak it if you need to
long offset = 0;
int read;
do
{
    read = dataStream.Read(buffer, 0, buffer.Length);
    Array.Copy(buffer, 0, data, offset, read);
    offset += read;
} while (read > 0);

I don't know what the List<byte> content is for, if you are just going to be moving data in to content be aware you may have some issues if response.ContentLength is larger than Int32.MaxLength because there are no members to pull out using the index or get the length via long instead of int.