openDocument: is an instance method of NSDocumentController.
You can subclass the NSDocumentController with your own. This will allow you to intercept openDocument:.
If you want to check the file before creating a document, you'll need to use NSOpenPanel for the open file dialog. Then call openDocumentWithContentsOfURL:display:completionHandler: when you want to create the document. If you don't want to create the document, you can trigger whatever you want to do instead.