Simple scraping of youtube xml to get a Python list of videos

Question

I have an xml feed, say:

http://gdata.youtube.com/feeds/api/videos/-/bass/fishing/

I want to get the list of hrefs for the videos:

 ['http://www.youtube.com/watch?v=aJvVkBcbFFY', 'ht....', ... ]

Answer 1

from xml.etree import cElementTree as ET
import urllib

def get_bass_fishing_URLs():
  results = []
  data = urllib.urlopen(
      'http://gdata.youtube.com/feeds/api/videos/-/bass/fishing/')
  tree = ET.parse(data)
  ns = '{http://www.w3.org/2005/Atom}'
  for entry in tree.findall(ns + 'entry'):
    for link in entry.findall(ns + 'link'):
      if link.get('rel') == 'alternate':
        results.append(link.get('href'))

as it appears that what you get are the so-called "alternate" links. The many small, possible variations if you want something slightly different, I hope, should be clear from the above code (plus the standard Python library docs for ElementTree).

Answer 2

Have a look at Universal Feed Parser, which is an open source RSS and Atom feed parser for Python.

Answer 3

In such a simple case, this should be enough:

import re, urllib2
request = urllib2.urlopen("http://gdata.youtube.com/feeds/api/videos/-/bass/fishing/")
text = request.read()
videos = re.findall("http:\/\/www\.youtube\.com\/watch\?v=[\w-]+", text)

If you want to do more complicated stuff, parsing the XML will be better suited than regular expressions

Answer 4

import urllib
from xml.dom import minidom
xmldoc = minidom.parse(urllib.urlopen('http://gdata.youtube.com/feeds/api/videos/-/bass/fishing/'))

links = xmldoc.getElementsByTagName('link')
hrefs = []
for links in link:
    if link.getAttribute('rel') == 'alternate':
        hrefs.append( link.getAttribute('href') )

hrefs