CFStreamCreateBoundPair streams lose data with small buffer size

Question

I am attempting to create a streaming html parser with libxml2 in Objective-C. I have a NSURLConnection that downloads the data, and I have created in NSInputStream and NSOutputstream with CFStreamCreateBoundPairwith a small buffer size of 10 bytes. As data is received from the NSURLConnection I write it to the output stream. It appears when the amount of data received is larger than the buffer size the left over data is lost. Is this supposed to happen? From my understanding I thought the data would be queued and written in chunks the size of the buffer to the input stream.

CFStreamCreateBoundPair Reference

Answer 1

You need to ensure that all data from the received chunk is eventually written into the stream.

You might do this with a simple loop in the delegate method where you continuously write a portion of the received chunk until it is completely written into the stream. However, this may cause the thread where the delegate is running to block undeterminably: if the consumer is not ready to consume more bytes, the output stream will block when attempting to write more data.

Alternatively, you might dispatch the NSData object asynchronously to a queue where a block is doing the loop and writes all the data before it completes. However, this may cause your system to run out of memory if the consumer is slow and the data is large - since all NSData live on the dispatch queue until the block finished.

Both approaches have pros and cons. I tend to prefer the first, since there is no memory issue and the connection will buffer the incoming bytes up to a certain upper limit anyway - before it stops acknowledging more bytes.