cURL dealing with large JSON-response

Question

I'm using the cURL-library with C++ to retrieve a very large JSON-string from this url: https://www.bitstamp.net/api/order_book/?group=1 (it is approx. 170 000 characters long)

My problem is that cURL doesn't pass the whole string to my callback function. I tested it with the standard callback function and it printed the whole string to stdout, so i can safely say that it's not a problem with receiving the data. But as soon as my callback function is called, the passed data has a size of either exactly 7793 or 15776 characters. (i can not think of any change that might correllate with the change of these 2 numbers)

I first thought the CURL_MAX_WRITE_SIZE value might be the cause so i changed it from 16384 to 131072 but it didn't help either:

#ifdef CURL_MAX_WRITE_SIZE
    #undef CURL_MAX_WRITE_SIZE
    #define CURL_MAX_WRITE_SIZE 131072
#endif

this is a brief overview of my code:

class EXCHANGE
{
   public:

   enum { BASE_TYPE }; //to check if template-types are derived from this class

   virtual size_t orderbookDataProcessing(void* buffer, unsigned int size, unsigned int nmemb) = 0; 

   template <typename TTYPE> //only for classes derived from EXCHANGE
   static size_t orderbookDataCallback(void* buffer, unsigned int size, unsigned int nmemb, void* userdata)
   {
       //check if TTYPE is derived from EXCHANGE (BASE_TYPE is declared in EXCHANGE)
       enum { IsItDerived = TTYPE::BASE_TYPE };   

       TTYPE* instance = (TTYPE*)userdata;

       if(0 != instance->orderbookDataProcessing(buffer, size, nmemb))
       {
           return 1;
       }
       else
       {
           return (size*nmemb);
       }
   }
}

class BITSTAMP : public EXCHANGE
{
   public:

   size_t orderbookDataProcessing(void* buffer, unsigned int size, unsigned int nmemb) 
   {   
       string recvd_string; //contains the received string

       //copy the received data into a new string
       recvd_string.append((const char*)buffer, size*nmemb);

       //do stuff with the recvd_string

       return 0;
   }
}


//main.cpp
///////////////////////////////////

CURL* orderbook_curl_handle = curl_easy_init();

curl_easy_setopt(orderbook_curl_handle, CURLOPT_URL, "https://www.bitstamp.net/api/order_book/?group=1") //URL to receive the get-request 

curl_easy_setopt(orderbook_curl_handle, CURLOPT_HTTPGET, 1) //use get method 

curl_easy_setopt(orderbook_curl_handle, CURLOPT_NOSIGNAL, 1)  //required for multi-threading ?

curl_easy_setopt(orderbook_curl_handle, CURLOPT_WRITEDATA, &bitstamp_USD)  //instance passed to the callback-function

curl_easy_setopt(orderbook_curl_handle, CURLOPT_WRITEFUNCTION, (size_t(*)(void*, unsigned int, unsigned int, void*))&EXCHANGE::orderbookDataCallback<BITSTAMP>) //callback function to recv the data  

curl_easy_setopt(orderbook_curl_handle, CURLOPT_SSL_VERIFYPEER, FALSE) //do not verify ca-certificate

curl_easy_perform(orderbook_curl_handle); 

Answer 1

CURL invokes your write callback with data as it receives it. If your data is too large (and yours is), it will be given to you via multiple invocations of the write callback.

You'll have to collect the data from each of the write callbacks, until CURL tells you that the response is complete. In your case, this should be when curl_easy_perform returns.