If it is possible, then implement the functionality in a command line tool, or another form of GUI-less application. For Cocoa applications it is possible to prevent them appearing on the Dock or in the Force Quit dialog, however a command line tool is a single binary file which does that anyway, so that would probably be a better way.
In terms of launching the tool, NSTask & NSPipe are your friends in this endeavour. The tool can definitely be kept inside your main Application's bundle, inside the Resources directory or some such, and then launched when needed. You can use the pipe to communicate back and forth.
I don't have any example code to hand, and its been a long while since I've had occasion to use either of these classes so the information I can give is limited, but it should be enough to point you in the right direction.