You can wire the sent action up to a method on your controller:
- (IBAction)cellDidEndEditing:(id)sender;
If you only want this method to be called when the return key is pressed, and not, eg, when the user clicks away, you can set the Action property on the NSTextField in IB to 'Send On Enter Only' instead of 'Send On End Editing'.
Within this method, you can access the event with:
NSEvent *evt = [NSApp currentEvent];
After checking whether the event is valid, you can obtain and act on the necessary modifier flags:
if (evt && evt.type == NSKeyDown) {
evt.modifierFlags // NSShiftKeyMask, NSCommandKeyMas, NSAlternateKeyMask
// Perform actions
}