Found the answer here: http://www.cocoabuilder.com/archive/cocoa/307917-validatemenuitem-not-called-for-undo-menu-item.html
Configuring the Undo menu item with action 'undo:' tells the frameworks that this is the undo menu item, so it becomes one element of a standard undo UI mechanism implemented in NSWindow. Under these circumstances, the menu item (and especially its validation) isn't yours to mess with.
If you want to take implement the entire mechanism yourself, then you should use a custom action such as 'myUndo:' for the Undo menu item. That means, as far as the frameworks know, you app doesn't have (standard) undo, and the action/validation should work in the normal way.