The Human Interface Guidelines state:
Avoid adding a submenu to an item in a pop-up menu. A submenu tends to hide choices too deeply and can be physically difficult for users to use.
I assume that since the practice is discouraged (albeit not that strongly) that it is not supported either. I can see how submenus would break some of NSPopUpButton's functionality, all of the item index related methods would not make sense. Those methods would have to use NSIndexPaths instead of simple indexes.
A potential substitute would be to use different indention levels in the menu to indicate hierarchy. I am about to try that myself.