New to Python - General Web Scraping

Question

In order to learn a bit more advanced Python, I tasked myself with creating a python script that navigates to a website (imdb.com, in this case), enters a word (which has already been declared as a variable), and then outputs the first 5 titles that come up into an array, which then outputs to the console.

My question is: Is something like this even possible? Are there libraries/frameworks that make this possible?

If it's possible, where would I start? Web scraping isn't new to me, but web scraping in Python is. All I really need is guidance towards the correct path. 25(ish) minutes of google searching came up with somewhat vague answers that only confused me more.

Answer 1

It is possible, you can use selenium to navigate trough the websites: http://docs.seleniumhq.org/ and to find the correct elements you can use XPath. There are good browser addons to test the XPaths.

Answer 2

You should definitely go the requests way. Making a request is as easy as:

import requests
r = requests.get('https://github.com/timeline.json')

(taken from requests' docs)

You simply have to find your site's URL of choice (http://www.imdb.com/find) and add the params ({'q': 'search_term'}) in the get method. Then you can access r.text and parse the results with a HTML parser (check BeautifulSoup). Storing the first 5 results and displaying them in the console should be a breeze.

Answer 3

u can U se the Third Party frame work caled Beautiful soup link and it easy to use

Beautiful Soup is a Python library designed for quick turnaround projects like screen-scraping. Three features make it powerful:

Beautiful Soup provides a few simple methods and Pythonic idioms for navigating, searching, and modifying a parse tree: a toolkit for dissecting a document and extracting what you need. It doesn't take much code to write an application Beautiful Soup automatically converts incoming documents to Unicode and outgoing documents to UTF-8. You don't have to think about encodings, unless the document doesn't specify an encoding and Beautiful Soup can't detect one. Then you just have to specify the original encoding. Beautiful Soup sits on top of popular Python parsers like lxml and html5lib, allowing you to try out different parsing strategies or trade speed for flexibility.

Answer 4

I strongly second the answer suggesting python requests, a lightweight solution for what you are trying to accomplish.

You can try something like:

import requests
r = requests.get(http://www.imdb.com/find?ref_=nv_sr_fn&q=liam&s=all)
return r.content

Looks like for imdb, you can alter the q= parameter in the url to return results. If I wanted X-Men instead of Liam, I can keep the same url except replace q=liam with q=xmen. For easier parsing, check out BeautifulSoup. If that's not your style, and if you want to get some regex practice, try using python regular expressions to pull the data you want.