URL fetch: prevent abuse, mailcious urls etc. in python/django

Question

I'm building a webpage featuring a very much a-like the facebook wall/newsfeed. Registered users (or through Facebook-connect, google auth) can submit urls. At the moment, I'm taking these URLs and use urllib2 to fetch the content of the URL and search for relevant information like og:properties, HTML title-tag and perheps some -tags for images.

Now, I understand that I'm putting my server at risk when I'm letting users feed my server with URLs to open.

My question is how high the risk is? What standard security checks can I make?

As for now, I am simply opening the url without any "active" protection because I don't know what to check for.

And what about storing fetched content into the database. Does django have built-in protection against SQL-injections?

Thank you!

Answer 1

One of the obvious risks here is that one could use your website as a vector for spreading malicious URLs.

E.g. Say I figure out a malformed html that allows for arbitrary code execution in webkit based browsers, say by exploiting a certain 0-day buffer overflow. Say your website goes popular, that'd be one of the spots I'd definitely try.

Now, you can't possibly match the contents of the URLs submitted to look for security flaws. You'd become an anti-virus/security company then. Both Chrome & Safari do take care of these to some extent.

For user's/content's sake and for the risk I explained, you could build in a flagging system that learns by user's actions. You could train a classifier whenever someone flags a URL, see examples here.

I'm sure there is a variety of such solutions, also in python.

For a quick overview of security, sql injections in Django's context, checkout this link.