Here's what I've come up with...
I took the code sample from C++ Windows Shell context menu handler (CppShellExtContextMenuHandler) and adjusted it considerably. I started by making it act on * and Directory, because in my case the user should be able to perform this action on any combination of files and folders. Next, instead of popping up a message box with the command, I changed it to use CreateProcess with a redirected stdin (as per the example here). Once the process is created, I write the list of files to the pipe, one per line. (I'm going to try changing that to write the list of files to the pipe before starting the process, to avoid the potential race issue.) My target application simply reads in the console line by line until it runs out of data, and uses that list of files.
It's a longer and more perilous process than I'd like - I crashed the shell a couple times during testing because of oversights in coding the extension! - but it gives me exactly what I want, which is the ability to right-click a set of files in Explorer, have them all open in the same instance of my app, and then right-click on a different set of files and have them open in a separate instance.
Of course, just because I have an answer doesn't mean it's the answer. I'm not going to mark this as the final answer for a little while, in case anyone else comes along with something better. Heck, even if you don't think your answer is better, I'd love to hear it! It's always fun to see how other people approach a problem.