You are just giving the CLR a hard time to locate the PDBs. The info you see in the debugger's Modules window is not relevant, that only matters for the debugger. The CLR has its own mechanism to locate and read PDBs. Necessarily so, this still needs to work without a debugger on the user's machine.
It is technically possible to tell the CLR where to look by setting an environment variable. Say you stored the PDB files in a subdirectory named "pdbs" then you could put this code in your Main() method:
var appdir = System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetEntryAssembly().Location);
var pdbdir = System.IO.Path.Combine(appdir, "pdbs");
Environment.SetEnvironmentVariable("_NT_SYMBOL_PATH", pdbdir);
Or just don't give it a hard time and put them in the same directory as your EXE.