tl;dr - this is a bug introduced in the December 2013 release of the AJAX Control Toolkit. I would strongly recommend moving away from the toolkit, and using something simpler and lighter - either hand-rolling things with jQuery, or using a framework like jQuery UI.
Or, you could possibly roll back to a previous version of the toolkit (use the links in the box on the right to view previous releases).
Looks like this is a bug in the AJAX Control Toolkit (or possibly in the Sizzle CSS Selector Engine, which is used by the toolkit).
You can actually repeat this error if you go to the AJAX Control Toolkit Sample site's Tab sample page. When I visit it in IE8, I get a similar error:
Notice, though, that the error is in "CombineScriptsHandler.axd". The file says the following right above the line that throws the error:
/*!
* Sizzle CSS Selector Engine
* Copyright 2012 jQuery Foundation and other contributors
* Released under the MIT license
* http://sizzlejs.com/
*/
And the specific statement that generates the error is here:
n._tabIndex=i.owner.get_tabs().length;
Of course, that's from the minified JS, so it's hard to say what it's doing.
The problem has also been reported on the Toolkit's codeplex page: