Wait for the browser to load COMPLETELY (or extract Captcha from website)

Question

I have a big problem and im trying to fix it for days and its just not working. I want to make a program that extracts a Captcha from a website and then show it to the user, the user solves it and then the programm checks if the code is right or wrong.

The problem is that the event "Webbrowser1_DocumenCompleted" doesn't wait untill the scripts are COMPLETELY loaded. So, sometimes the code shows, but and sometimes not.. its like a 50/50 chance..

Here is my code so far:

Imports System.Text
Imports System.Text
Public Class Form1

Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
    WebBrowser1.Navigate("http://minecraft-server.eu/vote/index/2421")
End Sub

Private Sub WebBrowser1_DocumentCompleted(ByVal sender As System.Object, ByVal e As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles WebBrowser1.DocumentCompleted

    Try
        Dim htmlDocument As HtmlDocument = Me.WebBrowser1.Document
        Dim htmlElementCollection As HtmlElementCollection = htmlDocument.Images
        Dim ImagesFound As Integer = 0
        For Each htmlElement As HtmlElement In htmlElementCollection
            Dim imgUrl As String = htmlElement.GetAttribute("src")
            If imgUrl.Contains("google.com/recaptcha/api/image?") Then
                ImagesFound += 1
                Select Case ImagesFound
                    Case 1
                        WebBrowser2.Navigate(imgUrl)
                End Select

            End If
        Next
    Catch ex As Exception
    End Try
End Sub

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
    WebBrowser1.Document.GetElementById("recaptcha_response_field").SetAttribute("value", tbText.Text)
    WebBrowser1.Document.GetElementById("minecraftname").SetAttribute("value", minecraftnamebox.text)
    WebBrowser1.Document.GetElementById("button").InvokeMember("click")
End Sub

Private Sub Button2_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button2.Click
    Dim sQuelltext As String = WebBrowser1.DocumentText.ToString

    If sQuelltext.Contains("Captcha Falsch") Then
        MsgBox("Wrong verification Code! Try again.")
        Application.Restart()
    Else
        End
    End If
End Sub

Private Sub Button3_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button3.Click
    Application.Restart()
End Sub
End Class

Answer 1

You problem is understandable. Because the browser has no clue that there's more stuff to be executed. When a script has finished loading, it doesn't mean that no more work to be done. It could be that a script waits 200 milliseconds and then does something, so there's no generic way of determining when a page has 'fully loaded'.

The DocumentCompleted event fires when the whole page has been downloaded, and all scripts, images and other dependencies have loaded. So, the DocumentCompleted does fire at the right time.

However, the captcha seems to be created some time after the document has completed loading, as said, the browser has no clue that he's supposed to wait on this. We can confirm this theory by visiting the page in our browser:

http://minecraft-server.eu/vote/index/2421

When the browser hides the spinning icon (which indicates loading) it takes about 500 more milliseconds for the captcha to actually be visible. You obviously can't rely on 500 milliseconds because it all depends on the connection.

The WebBrowser component does expose the FileDownload event, which occurs everytime a file is downloaded. This could be an image that's on the web page, anything. The only problem is that when the event fires, there's no way of know what file the event fired for.

The easiest solution in this case is to check whether the captcha has been loaded, each time the FileDownload event fires. You should prevent checking for the captcha in the FileDownload event before the document was completely loaded. So, before iterating over all HTML elements, check whether the document was completely loaded.

Answer 2

Try checking if WebBrowser1.ReadyState equals WebBrowserReadyState.Complete.