I wrote to the author of Colorbox, and he tracked the issue down to the following lines:
if (document.addEventListener) {
document.addEventListener('focus', //trapFocus, true);
$events.one(event_closed, function (){
document.removeEventListener('focus', trapFocus, true);
;}
I outcommented those, and now it seems to work (don't know about older IE versions and other problems though).