This might be just a try to get to the problem, not an answer, but stackoverflow doesn't let me comment because of my reputation.
Anyway, I remember to had weird behavior of NSOutlineView in the past. Maybe the setFloatsGroupRows: will help you:
[myOutlineView setFloatsGroupRows:NO];
Please let me know.