Fake a cookie to scrape a site in python

StackOverflow https://stackoverflow.com/questions/1117491

  •  12-09-2019
  •  | 
  •  

Question

The site that I'm trying to scrape uses js to create a cookie. What I was thinking was that I can create a cookie in python and then use that cookie to scrape the site. However, I don't know any way of doing that. Does anybody have any ideas?

Was it helpful?

Solution

Please see Python httplib2 - Handling Cookies in HTTP Form Posts for an example of adding a cookie to a request.

I often need to automate tasks in web based applications. I like to do this at the protocol level by simulating a real user's interactions via HTTP. Python comes with two built-in modules for this: urllib (higher level Web interface) and httplib (lower level HTTP interface).

OTHER TIPS

If you want to do more involved browser emulation (including setting cookies) take a look at mechanize. It's simulation capabilities are almost complete (no Javascript support unfortunately): I've used it to build several scrapers with much success.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top