Question

How to assign to workers a proxy that requires user name - password and a custom user agent using Selenium, PhantomJS driver with Python bindings.

I've had good success with creating many workers traversing my test website. I can also assign a user agent or a proxy that does not require authorization. But I haven’t figured out how to do both to the same worker at the same time yet.

However the real issue at the moment is assigning a proxy to the workers that require authorization by a user name and password.

The Players: Selenium 2.33.0 / PhantomJS 1.9.1 / Python 2.7.3 / Ubuntu 12.04

Me: Nube. Python weeks, Linux days, Selenium hours, PhantomJS -= , SO first post

Searches Yielded: How do I set a proxy for phantomjs/ghostdriver in python webdriver?

The answers may in fact be there and many other places I have read and re-read, but I can’t connect the dots at my present level.

User Agent solved with this method.

dcap = dict(DesiredCapabilities.PHANTOMJS)           
dcap["phantomjs.page.settings.userAgent"] = (
                "Any User Agent string here”)
driver = webdriver.PhantomJS(desired_capabilities=dcap)

Proxy without authorization works with this:

service_args = [
    '--proxy=127.0.0.1:9999',
    '--proxy-type=http,
    ]
driver = webdriver.PhantomJS('/usr/local/bin/phantomjs,service_args=service_args)

If both above methods are used I’m unsure how to pass both proxy and UA to the PhantomJS driver. ATM I’m only able to do one or the other and not at all with a proxy that requires authorization.

Goal for this SO thread:

  1. Learn how to assign a proxy that requires user name / password
  2. Assign a custom user agent to the same worker.

Using Selenium, PhantomJS driver with Python bindings.

The end game goal is to assign each worker a unique ip and pull from a pool of user agents. Creating the logic for this I remain optimistic but the proxy with authorization is kicking me at the moment.

As you can tell I’m very new to all of this and would appreciate any help and examples to this particular problem.

Thanks!

EDIT: Below accepted answer is incorrect. Unable to reproduce below solution. Only the proxy with authorization is assigned to the driver. Still unable to assign both proxy and a user agent to the same driver.

Any help or direction would be greatly appreciated.

EDIT.02: Issue resolved. It was never a coding issue. A new proxy provider at the server level assigned a default UA that overrode the above script. Once this was removed all was good.

Was it helpful?

Solution

Assign User Agent by Desired Capabilities

dcap = dict(DesiredCapabilities.PHANTOMJS)           
dcap["phantomjs.page.settings.userAgent"] = (
    "Your User Agent String here . . .")

Found API Reference here for the proxy authorization.
Add "--proxy-auth=username:password" to server_args. Like . . .

service_args = [
    '--proxy=xxx.xxx.xx.xxx:xxxx',
    '--proxy-auth=username:password',
    '--proxy-type=http',
    ] 

Then use both when starting the webdriver

driver = webdriver.PhantomJS(desired_capabilities=dcap,service_args=service_args)

This took care of all my issues.

EDIT: Unable to reproduce solution. Only proxy is changed with above method.

EDIT.02: Issue resolved. It was never a coding issue. A new proxy provider at the server level assigned a default UA that overrode the above script. Once this was removed all was good.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top