I found a solution using an undeclared method (not ideal, obviously). First declare it as follows:
@interface NSMenu (secret)
- (void) _setHasPadding: (BOOL) pad onEdge: (int) whatEdge;
@end
Then, sometime before showing the menu, do:
if ([menu respondsToSelector: @selector(_setHasPadding:onEdge:)])
{
[menu _setHasPadding: NO onEdge: 1];
[menu _setHasPadding: NO onEdge: 3];
}
My testing shows that this doesn't work on Snow Leopard, but does work on Lion, Mountain Lion, and Mavericks.