Scraping delayed ajax with Selenium / Splinter

Question

I am trying to write a script to identify if potential new homes have Verizon FiOS service available.

Unfortunately the site's extensive use of javascript has prevented everything from working. I'm using selenium (wrapped in the splinter module) to let the javascript execute, but I can't get past the second page.

Here is a simplified version of the script:

from splinter import Browser

browser = Browser()
browser.visit('https://www.verizon.com/FORYOURHOME/ORDERING/OrderNew/OrderAddressInfo.aspx')

nameAddress1 = "ctl00$body_content$txtAddress"
nameZip = "ctl00$body_content$txtZip"
formFill = {nameAddress1: '46 Demarest Ave',
            nameZip: '10956'}
browser.fill_form(formFill)
browser.find_by_id('btnContinue').first.click()

if browser.is_element_present_by_id('rdoAddressOption0', wait_time=10):
    browser.find_by_id('rdoAddressOption0').first.click()
    browser.find_by_id('body_content_btnContinue').first.click()

In this example, it chooses the first option when it asks for confirmation of address.

It errors out with an ElementNotVisibleException. If I remove the is_element_present check, it errors out because it cannot find the element. The element is visible and clickable in the live browser that selenium is controlling, but it seems like selenium is not seeing an updated version of the page HTML.

As an alternative, I thought I might be able to do the POST request and process the response with requests or mechanize, but there is some kind of funny redirect that I can't wrap my head around.

How do I either get selenium to behave, or bypass the javascript/ajax and do it with GET and POST requests?

Answer 1

The problem is that the input you are clicking on is really hidden by setting display: none style.

To workaround it, execute javascript code to click on the input and set checked attribute:

browser.execute_script("""var element = document.getElementById('rdoAddressOption0');
                          element.click();
                          element.checked=true;""")
browser.find_by_id('body_content_btnContinue').first.click()