How to get a specific link for web scraping

Question

I'm writing a program to get the market value of soccer players and it works

from bs4 import *
import requests

def marketvalue():

reply = str(input("Enter the player whose current market value you wish to know \n"))

url = ("http://www.transfermarkt.com/en/first-last/profil/playerid.html")

r = requests.get(url)

data = r.text

soup = BeautifulSoup(data)

mydivs = soup.findAll("td", {"class" : "note"})

value = str(mydivs[0].get_text())

print("Current value of {0} is {1}".format(reply, value))

The problem is that the URL for the player is based on the playerid which is seemingly random. Is there anyway to get this using requests or soup?

Answer 1

import mechinize http://wwwsearch.sourceforge.net/mechanize/documentation.html Using this along with BeautifulSoup should provide you with the results you are looking for. More Docs. http://www.ibm.com/developerworks/web/library/l-python-mechanize-beautiful-soup/index.html?ca=dat