headless browser or browser control with ajax/javascript/proxy and multithread support?

Question

I've looked around on lots of messages here but am yet to find one browser control or headless browser that allows me to multithread and configure proxies per thread as well as supporting ajax/javascripts.

I've tried to far:

• awesomium - works pretty well if I made a separated application to spawn for each thread I need to run but not what I am looking for.
• simplebrowser - works well but the js/ajax manipulation is too weak to consider
• nhtmlunit - works far too complicated to get up and running on c# and doesnt work well
• watin - can't instantiaty private session and cant have proxy per instance

I've tried others too but these were the main ones I recall the tests.

If awesomium was able to do multithread that would be a awesome but I heard they will be only doing that on further releases, it was the best one so far.

Could anyone recommend me other for testing or that are capable of doing the above asked ?

My main goal here is to automate tests on a local network website.

UPDATE:

Correct me if I am wrong, but unlike watin, selenium webdriver(will try and test it in a few) seems to need a remote server running aside from the application itself to control the browser, still looking for alternatives.

UPDATE2:

Have been able to do multithread with chromiumdriver + Selenium and c#, however it uses an incridible amount of memmory for just a few threads on top of spawning multiple cromiumdrivers + browsers:

Fire multiple threads:

_tasks = new List<Task>();
foreach (ProxyList item in _proxy)
{
            // using it directly it doesnt get passed down to the function
    string proxy = item.Proxy;
    string host = item.Host;
    _tasks.Add(Task.Factory.StartNew(() =>
    {
        ChromeGetIp(proxy, host);
    }));
}

Chrome

private void ChromeGetIp(string proxy, string host)
{
    string dir = Application.StartupPath + "\\" + host;
    ChromeOptions profile = new ChromeOptions();
    profile.AddArguments(new string[] { "-proxy-server=" + proxy, "-incognito", "--new-window", "-user-data-dir=" + dir });
    IWebDriver driver = new ChromeDriver(profile);
    driver.Navigate().GoToUrl("http://checkip.dyndns.com/");
    if (driver.PageSource.Contains("Current IP Address"))
        MessageBox.Show(driver.FindElement(By.TagName("body")).Text, host);
    else
        MessageBox.Show("Failed", host);
    driver.Quit();
}

Is other ways to make this lighter to be worth using for several threads ?

Answer 1

So far the best solution I have found was using awesomium, it uses very little memory when used headless, allowing me to spawn lots of processes, compared to selenium and others solution and have a good support for AJAX, JavaScript, proxy, and more.

For the time being I will leave my own answer marked however if anyone posts a better option that meets my requirements I will change it.

Answer 2

I'm looking for a similar solution, finding driving automation using headless HttpWebRequest next to impossible for anything with JS. Ie, 99% of the internet.

Anyway - Selenium does not need the server, unless you want to do things remotely. If you're looking to write some C# to drive your browser, on the same machine - you're good to go with Selenium.

Explains it here: http://seleniumhq.org/docs/03_webdriver.html

Edit: Had to edit this answer, as I cant comment (wasnt logged in when I answered).

If you want to do headless browsing - or at least where the browser isn't visible, you can set it up as a service. That will execute as System and the browsers will not be visible to you. It DOES get memory heavy, but unless you're going to run 100 browsers at once, it's really no problem for any modern computer. And be sure to Quit() the browsers when necessary/when service ends.

As for the proxy - you can't set it with IE, but Firefox will take it in with Capabilities, which is what Im doing.

Answer 3

Any reason to reject the usual Selenium/Firefox combination?