I think you can accomplish what you want by subclassing NSButton and implementing
- (void)mouseDown:(NSEvent *)theEvent
- (void)mouseUp:(NSEvent *)theEvent
and your own method to set the inactive state.
These methods would call
- (void)setImage:(NSImage *)anImage
To change from active to pressed to inactive states using the different images.
You also have to uncheck "Bordered" in Interface Builder on your NSButton to stop the button background from showing.
Also, calling
- (void)setEnabled:(BOOL)enabled
on the NSToolbarItem will change the palette label to active/inactive (grey the text below the button).