Clicking link using beautifulsoup in python

Question

In mechanize we click links either by using follow_link or click_link. Is there a similar kind of thing in beautiful soup to click a link on a web page?

Answer 1

BeautifulSoup is an HTML parser.

Further discussion really depends on the concrete situation you are in and the complexity of the particular web page.

If you need to interact with a web-page: submit forms, click buttons, scroll etc - you need to use a tool that utilizes a real browser, like selenium.

In certain situations, for example, if there is no javascript involved in submitting a form, mechanize would also work for you.

And, sometimes you can handle it by simply following the link with urllib2 or requests.

Answer 2

print(soup.find('h1',class_='pdp_product_title')) does not give any result <div class="pr2-sm css-1ou6bb2"><h2 class="headline-5-small pb1-sm d-sm-ib css-1ppcdci" data-test="product-sub-title">Women's Shoe</h2><h1 id="pdp_product_title" class="headline-2 css-zis9ta" data-test="product-title">Nike Air Force 1 Shadow</h1></div>