The simple subtle rule being broken here is that the PDB file name appears to be encoded into the EXE file, not derived from the EXE file name. So, for example, if you create "Copy of myProgram.exe" and "Copy of myProgram.pdb" by a simple copy-and-paste in Windows Explorer, it can appear that symbols are being loaded despite the name change. But this will only work so long as the original "myProgram.pdb" file is also present - if you remove "myProgram.pdb" and only have "Copy of myProgram.pdb", the symbols may be compatible, but Visual Studio will not even try to load them.
In a quick-and-dirty test on a remote machine in my office, all that changed was the file name, not the file version, whereas on the actual remote machine, both name and file version were different, so it was never possible for things to appear to work.