On Linux, change the XDG file associations. You'll need to first associate the extension with a MIME type, then register a .desktop
file to open that MIME type.
On OS X, you have to put the Python program in an application bundle, and then add the file extension to the list of extensions which that program handles. This is done by editing the bundle's Info.plist
.
On Windows, you have to change registry settings. Here is how to do it in WiX, if you use WiX to write your installer.
Summary: Probably best to just use an "open file" dialog within your application, or use a command-line interface, or something simpler.