HTTP in C: Why do I have to call recv() multiple times?

Question

I've started to learn about Network programming in C (sockets), and Internet protocols.

Currently I'm focusing on HTTP.

During my (very) short experience with implementing HTTP client using C, I discovered that sometimes you have to call recv() multiple times in order to get the entire message from the server - even though the client didn't send a message to the server between those seperate calls to recv().

For example:

When I was trying to implement a HTTP client with one call of recv(), I got just the headers (I think, I'm still a newbie with HTTP). But when I called recv() twice, I got the headers in the first call, and the body (the html code) in the second call.

It is not a problem of a short buffer, because the buffer I was using was long enough to hold the entire message.

Why does it happen? What is the reason that the client has to call recv() multiple times, although there is no new data that was sent by the client? I thought that if the client doesn't send new data to the server, the call to recv() will bring the entire response of the server.

I don't think this is a problem with my code, but if you are going to ask for the code, I have no problem to post it here. I just think this is unnecessary, correct me if I was wrong.

I don't think this is relevant, but I'm using Winsock2.

Answer 1

TCP was designed to be stream oriented. The receiver doesn't know anything about sent messages/packets. All it sees is a simple stream of bytes.

So you think like "I sent a message", but recv() only received half of it. The fact is, you never sent a message. You sent a bunch of bytes, and you cannot expect all of them to be received by a single call. You could receive them in 1, 2 or many calls; you could receive the end of a previous "message" combined with the beginning of the next one and so on.

The only guarantees are that you will receive the bytes in the same order they were sent and you will never receive 0 bytes until the stream is closed. That's how the network API works and you have to get used to it.

Answer 2

The receiver does not know anything how much send calls and how much bytes in each call
are made at the sender. You can´t say that you want a recv with 10 byte
for each send with 10 byte.
A recv call could receive the data of three sends,
or the data of one send could be split over three receives...
If they match sometimes, it is pure coincidence.

Answer 3

One thing you can do is to use a loop like this.

    while((recv(socketi,response,sizeof(response)-1,0))>0){
    }
    printf("\n%s",response);

In the request you had send, you had to specifically say that you are using HTTP/1.0 GET / HTTP/1.0 (to close the connection after you receive all the data). Otherwise it will wait until a timeout occurs.

By doing so you will get all the response in the array. No need to call recv() multiple times.