The technique is known as protocol activation (JavaScript reference | C#/VB/C++ reference) and you set most of this up in the manifest (see image below). If there is only one application that's installed and set up to handle protocol "XYZ" then that app will launch for the given URI; if there are multiple one that can handle it, then the user is prompted for which app they would like to launch.
Jerry Nixon has a great walkthrough on his blog as well.