I'm at least able to partly answer my own question after having looked at Apple's iSpend sample application. Their file TransactionsController_Sorting.m includes a method scheduleRearrangeObjects that invokes rearrangeObjects in a different way. Changing my own code in the same way means including this snippet in the setName: method:
#ifdef REARRANGE_FROM_SETNAME
// Commented out: [treeController rearrangeObjects];
[treeController performSelector:@selector(rearrangeObjects) withObject:nil afterDelay:0.0];
#endif
With this change, the outline view no longer loses focus after renaming a node. What's left to do now is take this code out of the model and into the view/controller; TransactionsController_Sorting seems to also illustrate how to do that. (I still don't understand why the above change prevents the outline view from losing focus though, anyone have an explanation?)