Everything is okay:
NSIndexSet *set = [[NSIndexSet WSSIndexSetFromMask:0] addIndex:0];
No visible @interface for 'NSIndexSet' declares the selector 'addIndex:'
instancetype
says to the sender, that you return a instance of the receivers type even it is a subtype. For the sender it is a NSIndexSet, because it is send to the class object of NSIndexSet.
An introspection that way, that someone looks to the return type and sees a subclass and takes any advantage out of this information, is malformed. The contract is build with the return type and this is in this case NSIndexSet.