How to get text from a web page using C++?

Question

I am trying to use Poco to grab news from the front page of reddit. I'm looking at this pdf ( http://pocoproject.org/slides/200-Network.pdf ) for the answer, but it's a bit over my head at this point and I'm not sure how to accomplish my goal. As I said, I'm trying to simply grab the news articles (specifically, the article titles) from www.reddit.com.

The code I have so far grabs ALL off the html from reddit's front page and cout's it to the screen:

#include <iostream>        
#include "Poco/Net/SocketAddress.h"
#include "Poco/Net/StreamSocket.h"
#include "Poco/Net/SocketStream.h"
#include "Poco/StreamCopier.h"

using namespace std;
using namespace Poco::Net;
using namespace Poco;

int main(int argc, char *argv[])
{   
    SocketAddress sa("www.reddit.com", 80);
    StreamSocket socket(sa);
    SocketStream str(socket);
    str << "GET / HTTP/1.1\r\n"
     "Host: www.reddit.com\r\n"
     "\r\n";
    str.flush();

    StreamCopier::copyStream(str, cout);

    system("PAUSE");
}

Looking at the above mentioned pdf, it looks like my answer may be in there somewhere, but I am still learning about computer networks and internet protocol, so most of it is above my head at this point.

Main Question: Can someone help me figure out how to get the article titles from www.reddit.com into a string or array of strings?

Answer 1

Why not grab http://www.reddit.com/.rss, which is much simpler than html? For example to get news titles using qt framework:

class Foo : public QObject { Q_OBJECT
public:
  Foo();
private slots:
  void got_it(QNetworkReply* reply);
private:
  QNetworkAccessManager* news_grabber;
};

Foo::Foo() {
  news_grabber = new QNetworkAccessManager(this);
  QObject::connect(news_grabber, SIGNAL(finished(QNetworkReply*)),
           this, SLOT(got_it(QNetworkReply*)));
  news_grabber->get(QNetworkRequest(QUrl("http://www.reddit.com/.rss")));
}

void Foo::got_it(QNetworkReply* reply) {
  QDomDocument document;
  std::vector<QString> items_storage;
  document.setContent(static_cast<QIODevice*>(reply));
  QDomNodeList items = document.elementsByTagName("item");
    for (int i = 0; i < items.length(); i++)
      items_storage.push_back(items.at(i).firstChildElement("title").text());
  }