OK, I finally figured out what the heck was going on here. It turns out the problem was that I was implementing the -splitView:constraintMinCoordinate:ofSubviewAt:
delegate method (as well as the maxCoordinate one) to restrict the size of the split subviews in the vertical direction. Yes, restricting the vertical resizing of the split view affected the horizontal layout of the buttons.
It appears that what happens is that, if you implement those delegate methods, NSSplitView reverts back to using autoresizing masks to layout the subviews rather than auto layout constraints. Since the view containing those buttons is no longer participating in auto layout, the buttons smush together when you resize the window small. In the simulator, the split view doesn't have a delegate set, so all the auto layout stuff works fine in that environment. Note that merely having the methods implemented is enough to trigger this, even if they just return the proposed coordinates unchanged.
The solution ended up being quite easy, which was to delete the delegate methods and replace it with a vertical constraint on the subview to restrict its size instead.