You can't directly query for elements that are overflowing as they are not hidden as far as the DOM is concerned (and there are no attribute changes to check for with JQuery).
You will need to check positions instead against the width/height of the menu.
JSFiddle here: http://jsfiddle.net/TrueBlueAussie/WFGJ4/
menu.find('li').each(function () {
totalWidth += $(this).outerWidth();
if (totalWidth > menuWidth) {
$(this).before("<li><a>More</a></li>");
return false; // Abort loop
}
});
I put auto scrolling on the menu so you can actually see what is overflowed.