Follow a link with Ghost.py

Question

I'm trying to use Ghost.py to do some web scraping. I'm trying to follow a link but the Ghost doesn't seem to actually evaluate the javascript and follow the link. My problem is that i'm in an HTTPS session and cannot use redirection. I've also looked at other options (like selenium) but I cannot install a browser on the machine that will run the script. I also have some javascript evaluation further so I cannot use mechanize.

Here's what I do...

## Open the website
page,resources = ghost.open('https://my.url.com/')

## Fill textboxes of the form (the form didn't have a name)
result, resources = ghost.set_field_value("input[name=UserName]", "myUser")
result, resources = ghost.set_field_value("input[name=Password]", "myPass")

## Submitting the form
result, resources = ghost.evaluate( "document.getElementsByClassName('loginform')[0].submit();", expect_loading=True)

## Print the link to make sure that's the one I want to follow
#result, resources = ghost.evaluate( "document.links[4].href")

## Click the link
result, resources = ghost.evaluate( "document.links[4].click()")

#print ghost.content

When I look at ghost.content, I'm still on the same page and result is empty. I noticed that when I add expect_loading=True when trying to evaluate the click, I get a timeout error.

When I try the to run the javascript in a Chrome Developper Tools console, I get

event.returnValue is deprecated. Please use the standard event.preventDefault() instead.

but the page does load up the linked url correctly.

Any ideas are welcome.

Charles

Answer 1

I think you are using the wrong methods for that.
If you want to submit the form there's a special method for that:

page, resources = ghost.fire_on("loginform", "submit", expect_loading=True)

Also there's a special ghost.py method for performing a click:

ghost.click('#some-selector')

Another possibilty, if you just want to open that link could be:

link_url = ghost.evaluate("document.links[4]")[0]
ghost.open(link_url)

You only have to find the right selectors for that.
I don't know on which page you want to perform the task, thus I can't fix your code. But I hope this will help you.