Codepage 850 works, 65001 fails! There is NO response to “call foo.cmd”. internal commands work fine

Question

The question basically explains the problem.

I'm using Windows XP Pro Service Pack 3
ComSpec=C:\WINDOWS\system32\cmd.exe
I launched the console via Start... Run-dialog... cmd.exe

Here is a "view" of my console:
The command, then the output (and my // comments)

C:\> chcp 850
Active code page: 850
// output is as expected

C:\> echo @chcp ^& REM 850>test850.cmd
// no output; as ecpected)

C:\> type test850.cmd
@chcp & REM 850
// output is as expected

C:\> call test850.cmd
Active code page: 850
// output is as expected

The above works fine (as expected). Things are happy in Windows-land, but the "call" FAILS when I switch to codepage 65001

C:\> chcp 65001
Active code page: 65001
// output is as expected

C:\> echo @chcp ^& REM 65001>test65001.cmd
// no output; as ecpected

C:\> type test65001.cmd
@chcp & REM 65001
// output is as expected

C:\> call test65001.cmd
// NO OUTPUT, NO ERROR, NO ANYTHING, NADA... other than frustration :)

What is happening (NOT happening) here?

Answer 1

Interesting, it's not actually running it at all. If you do the following:

pax> echo echo yy >xx.cmd
pax> chcp 850
pax> xx
yy
pax> chcp 65001
pax> xx
pax> _

you get nothing. It's not just missing output, it's not running at all (as evidenced by putting start . before the echo line). In code page 850, Explorer runs, not so for code page 65001.

There's some discussion on the issue over here. You can get your script to run with:

chcp 65001 && xx.cmd && chcp 850

so it seems to be some sort of problem in starting command files but only when the code page is 65001 before you enter the command!

Others on the net seem to be suggesting that PowerShell may be a good choice, given the finicky support in cmd.exe. That's a decision you'll have to evaluate for yourself but, working in a large organisation myself with many tools to do the same job, I suspect Microsoft will be placing any enhancement efforts behind PowerShell rather the older command shell. Their resources are large but not unlimited.

Answer 2

The cause of this is when cmd.exe for windows xp internally call to the function MultiByteToWideChar using the argument dwFlags with the value 1. The documentation says this: "For UTF-8 dwFlags must be set to 0. Otherwise, the function fails".

A patch for it here: http://consolesoft.com/p/cmd-xp-65001-fix/index.html