You're probably looking for functionality that is defined in the NSApplicationDelegate-protocol. Specifically the following two methods give you some control over what happens on app start:
- (BOOL)applicationShouldHandleReopen:(NSApplication *)theApplication hasVisibleWindows:(BOOL)flag
- (BOOL)applicationShouldOpenUntitledFile:(NSApplication *)sender
See the documentation for more.