QFile saves files on program quit, wont save ~500MB files at all

Question

Yeah guys, it's me again. I have following code:

void MainWindow::on_startButton_clicked()
{
  QNetworkAccessManager *nam = new QNetworkAccessManager(this);
  QNetworkReply *re = nam->get(QNetworkRequest(QUrl("http://somesite/ai/iai.jpg")));
  QEventLoop loop;
  QObject::connect(reply, SIGNAL(readyRead()), &loop, SLOT(quit()));
  loop.exec();
  ui->dbgOut->insertHtml("<font color='green'>OK</font><br>");
  ui->dbgOut->insertHtml("##################################");
  //save
  QFile file("C:\\a.bin");
  file.open(QIODevice::WriteOnly);
  file.write(re->readAll());
  file.close();
}

And I have two problems:

1. When i click on button it downloads files but won't write them to hdd. I can wait 5, 10 minutes and nothing. At this time, whole file is stored inside program memory. If I close my program, they are saved on disk.

2. Large files (~500MB) aren't saved at all. When I close my program, it crashes instantly.

How I can edit it so my program will save downloaded files "in realtime"?

Answer 1

The non-interactivity is because re->readAll() on a device with unknown size is a blocking call. It will keep reading until the request is finished.

The problem with large files has to do with growing of the byte array that holds the file. At some point, your byte array will be, say 400MB, and then it will have to grow to say 2x that size, so you'll have to hold ~1GB at once, in two big chunks, and due to address space fragmentation an allocation request will fail and your program crashes.

Small changes to your code yield desired behavior: you start reading and writing at once, and just link the two:

class MainWindow {
  ...
  // You shouldn't be creating a new network access manager for each request.
  QScopedPointer<QNetworkAccessManager> m_nam;
  ...
};

void MainWindow::handleError(const QNetworkReply *) { ... }
void MainWindow::handleError(const QFile *) { ... }
void MainWindow::on_startButton_clicked()
{
  // Lazily create the network access manager once.
  if (!m_nam) m_nam.reset(new QNetworkAccessManager);
  // The file is destructed, and thus flushed and closed, when the
  // last shared pointer to this file is destructed.
  QSharedPointer<QFile> output(new QFile("C:\\a.bin"));
  if (!output->open(QIODevice::WriteOnly)) return;
  QNetworkReply *reply = m_nam->get(QNetworkRequest(QUrl("http://somesite/ai/iai.jpg")));
  // The lambda syntax creates a functor object that holds a copy
  // of the reply pointer and the output file pointer.
  // This functor will be destructed when the reply is destructed.
  QObject::connect(reply, &QIODevice::readyRead, [this, output, reply]{
    QByteArray data(reply->bytesAvailable(), Qt::Uninitialized);
    qint64 in = reply->read(data.data(), data.size());
    qint64 out = output->write(in);
    if (in < 0) {
      handleError(reply); 
      reply->deleteLater();
    }
    else if (out != in) {
      handleError(output.data());
      reply->deleteLater();
    }
  });
  // The reply will self-destruct when finished, thus deleting the file
  // instance.
  QObject::connect(reply, &QNetworkReply::finished, reply, &QObject::deleteLater);
}