Sending an xml message in parts through a TCP socket using Qt

Question

We are writing a project where there is a client who generates xml requests, sends them to a server, who parses the request and returns the requested information in an xml string.

The application works fine when the xml replies are small, but when they exceed about 2500 characters they sometimes get cut off on the client end. I say sometimes because when the client and server run on the same machine and communicate through the home address 127.0.0.1 the replies are parsed just fine. However, when the client and server are on different machines and communicate over lan, this is when the client cuts the message to around 2500 characters.

The communication is done by tcp sockets. We are using Qt, the client has a qTCPsocket, and the server a qTCPserver and a pointer to a qtcpsocket.

We think a possible solution to our issue is sending the xml in pieces, either separated by character count or by tag. While it is easy for us to break the message into parts, sending the parts and having the client or server read and compile the parts into one xml request is causing us trouble.

For the sake of example, we wanted to test having the client send a request in multiple parts.

Here is our client function call to send a request. xmlReq is generated else where and passed in. As an example of breaking the message into parts we strip away the closing tag from the xml request, and then send it as another piece later.

QString ClientConnection::sendRequest(QString xmlReq)
{

    this->xmlRequest = xmlReq;

    QHostAddress addr(address);

    QList<QString> messagePieces;
    xmlRequest.remove("</message>");

    messagePieces.append(xmlRequest);
    messagePieces.append("</message>");


    client.connectToHost(addr,6789);

    if(client.waitForConnected(30000))
    {
        for(int i = 0; i < messagePieces.length();i++)
        {  
            client.write(messagePieces[i].toAscii(),messagePieces[i].length()+1);
            qDebug() << "Wrote: " << messagePieces[i];
        }
    }


    char message[30000] = {0};

    xmlReply = "";

    if(client.waitForReadyRead(30000)){


        client.read(message,client.bytesAvailable());


    }else{
        xmlReply = "Server Timeout";
    }

    client.close();
    xmlReply = (QString) message;

    return xmlReply;

}

Next is our server code. It is written in such away that it is supposed to read messages from the client until it sees the xml closing message tag, and then process the data and send the reply back to the client.

This is the code that starts the server.

//Start the server, pass it the handler so it can perform queries
    connect(&server, SIGNAL(newConnection()), this, SLOT(acceptConnection()));
    server.listen(QHostAddress::Any, 6789);

When it gets a new connection it calls the acceptConnection slot which looks like this

    void CETServer::acceptConnection()
{
    client = server.nextPendingConnection();
    connect(client, SIGNAL(readyRead()), this, SLOT(startRead()));
}

The startRead looks like this:

void CETServer::startRead()
{
    char buffer[1024*30] = {0};

    client->read(buffer, client->bytesAvailable());

    QString readIn;

    readIn = (QString) buffer;

    ui->statusText->appendPlainText("Received: " + readIn);

    //New messages in will be opened with the xml version tag
    //if we receive said tag we need to clear our query
    if (readIn.contains("<?xml version =\"1.0\"?>",Qt::CaseSensitive))
    {
        xmlQuery = "";
    }

    //add the line received to the query string
    xmlQuery += readIn;

    //if we have the clsoe message tag in our query it is tiem to do stuf with the query
    if(xmlQuery.contains("</message>"))
    {
        //do stuff with query

        ui->statusText->appendPlainText("Query received:" + xmlQuery);

        QString reply = this->sqLite->queryDatabase(xmlQuery);
        xmlQuery = "";

        this->commandStatus(reply);

        if(client->isWritable()){
            //write to client
            client->write(reply.toAscii(),reply.length()+1);
            ui->statusText->appendPlainText("Sent to client: " + reply);
            client->close();

        }
    }}

In my mind, the start read is coded in such a way that anytime the client writes a message, the server reads it and attaches it to the xmlRequest the server stores. If the message contains , the xml closing tag, then it processes the request.

What happens though, is if the client does successive writes, the server does not read them all, only the first, and never receives the xml closing tag, and therefore processes no requests.

The question I need answered is why doesn't the server respond to the clients multiple writes? How should I go about making it such that I can send an xml string, broken down in pieces, and have the server read all the pieces and turn it into one string again?

Answer 1

That is happening because of the "stream" nature of the TCP protocol. The data is split into many packets, and in your app, you're really reading only a part of them (bytesAvailable() is not necessary equal to the amount of bytes the other host sent, it's just how many bytes are available in the socket buffer). What you have to do is to establish a simple protocol between the client and server. For example, the client first sends the STX character, then the XML, then ETX character. When the server sees an STX character, it reads everything into the buffer up until the EXT character. Other approach - send a 4-byte integer in network byte order indicating the size of XML data in bytes, then send XML. The other host must receive the integer, convert it to its native byteorder, then read the specified amount of data from the socket to the buffer.

Answer 2

In TCP, there is something known as the Maximum Segment Size. Before initialization of data transfer, both parties decide the MSS in the SYN handshake phase. That is the reason why your data is being split up.

You have only one client.read() . The server will send a reply for every read processed. You need a similar mechanism on the client side to handle reads. A function that reads till it has read N number of bytes. You can send the value N at the start of your data transfer.

Answer 3

COMP 3004 I see. Such a nightmare, we've been trying with QXmlStreamReader and QXmlStreamWriter. The writer is nice and simple, but the reader is a nightmare, we've been trying to use the PrematureEndOfDocument error as a break point for knowing there is more data.