There is no concept of a "rootViewController" in OS X applications. This is because applications are made up of one or more windows and/or a menu bar, none of which is the "root".
You can, however, find the window controller starting from a view controller:
[[[self view] window] delegate];
If you are looking to call custom methods on the window controller, it is probably better to create a delegate so that assumptions about the controller don't get you into trouble.
The equivalent of NSViewController.alloc.initWithNibName(nil, bundle: nil)
would be:
[[NSViewController alloc] initWithNibName:nil bundle:nil];
Method calls get nested in square brackets, methods with multiple parameters are simply label1:parameter1 label2:parameter2...
Usually, though, you would have a custom NSViewController
subclass that you would instantiate instead.