boost::asio::async_write and buffers over 65536 bytes

Question

I have a very simple method, with the purpose of responding to an incoming message, and then closing the connection:

void respond ( const std::string message )
{ 
  std::string str = "<?xml version=\"1.0\"?>";

  Controller & controller = Controller::Singleton(); 
  if ( auto m = handleNewMessage( _message ) )
  {
    auto reply = controller.FIFO( m );
    str.append( reply );
  }
  else
    str.append ( "<Error/>" );

  std::size_t bytes = str.size() * sizeof( std::string::value_type );
  std::cout << "Reply bytesize " << bytes << std::endl;

  boost::asio::async_write(
            socket_,
            boost::asio::buffer( str ),
            boost::bind(
                    &TCPConnection::handle_write,
                    shared_from_this(),
                    boost::asio::placeholders::error,
                    boost::asio::placeholders::bytes_transferred
                  ));
}


void handle_write ( const boost::system::error_code & error, size_t bytes_transferred )
{
  if ( error )
  {
    std::cerr << "handle_write Error: " << error.message() << std::endl;
    std::cerr << "handle_write Bytes sent: " << bytes_transferred << std::endl;
  }
  else
  {
      std::cerr << "handle_write Bytes sent: " << bytes_transferred << std::endl;
      socket_.close();
  }
}

I know the problem is that boost::asio::async_write does not complete the writing operation, because the output from the above operations is:

Reply bytesize: 354275
handle_write Bytes sent: 65536

Implying that the maximum buffer size (65536) was not enough to write the data?

Searching around Stack Overflow, I discovered that my problem is that the buffer created by the method:

boost::asio::buffer( str )

goes out of scope before the operation has a chance to finish sending all the data.

It seems like I can't use a boost::asio::mutable_buffer, but only a boost::asio::streambuf

Furthermore and more importantly, a second error complains about the actual boost::asio::async_write being passed a boost::asio::const_buffer OR boost::asio::mutable_buffer:

/usr/include/boost/asio/detail/consuming_buffers.hpp:164:5: error: no type named ‘const_iterator’ in ‘class boost::asio::mutable_buffer’
     const_iterator;
     ^
/usr/include/boost/asio/detail/consuming_buffers.hpp:261:36: error: no type named ‘const_iterator’ in ‘class boost::asio::mutable_buffer’
   typename Buffers::const_iterator begin_remainder_;

So I am left with only one choice: To use a boost::asio::streambuf

I've tried using:

  boost::asio::streambuf _out_buffer;

As a class member, and then made method respond:

  std::ostream os( &_out_buffer );
  os << str;

  boost::asio::async_write(
            socket_,
            _out_buffer,
            boost::asio::transfer_exactly( bytes ),
            boost::bind(
                    &TCPConnection::handle_write,
                    shared_from_this(),
                    boost::asio::placeholders::error,
                    boost::asio::placeholders::bytes_transferred
                  ));

However, although I get no errors, not the entire data is sent! So, I am guessing, not the entire string is written into the streambuf?

Alternatively, I would love to know what is the most elegant way to write using boost::asio::async_write, data that is larger than 65536 bytes!

Answer 1

Alex, you understand asio async operations wrong. Your problem is all about lifetime of buffer and socket. The buffer has to be alive and socket opened during the whole transmition time (from asio::async_write call to handle_write callback is to be called by Asio io_service dispatcher. To better understand how it works, consider that every time you doing some boost::asio::async_{operation} you are posting the pointer to data for operation and pointer to callback function to the job queue. And it's Asio decision when to execute your job (but of course it tries to do it as faster as possible =)). When the whole (possible big) I/O operation completes the Asio informs you using specified callback. And you can release resources then freely.

So, to get your code work you have to ensure that std::string str is still exist and _socket not closed until the handle_write callback. You can replace the stack allocated std::string str variable by some member variable in the class that agregates _socket. And move the socket_.close(); line from respond function to handle_write.

Hope, I helped you.

P.S. When you do boost::asio::buffer( str ), you don't copy content of the string but just create thin wpapper above data of string.

Answer 2

The code:

_out_buffer( static_cast<void*>( &str.front() ), bytes );

Is only valid when initializing _out_buffer, i.e. Before the body of your class's constructor begins.

That code equivalent to

_out_buffer.operator()( static_cast<void*>(&str.front()), bytes )

Of course there is no such operator in class mutable_buffer, and that's what the compiler is complaining about.

I think the simplest thing to do (but not the best), is to change that line to:

_out_buffer = boost::asio::mutable_buffer(
    static_cast<void*>( &str.front() ),
    bytes
);