There are a couple of problems.
First of all you are using self-signed certificate, so you should define it explicitly by adding -r key to makecert command or you'll get an error "The signer's certificate is not valid for signing" at sign step.
Next, at this step
signtool.exe sign /f "App-O.pfx" /p fess "C:\Output\setup.exe"
you are trying to open pfx using password "fess". But you actually didn't set any password for pfx file. To do it you should add -po key to pfx creation command.
After that you can sign your application.
So the correct process will be:
makecert.exe -sv App-O.pvk -n "CN=MY DIGITAL KEY" App-O.cer -r
pvk2pfx.exe -pvk App-O.pvk -spc App-O.cer -pfx App-O.pfx -po fess
signtool.exe sign /f "App-O.pfx" /p fess "C:\Output\setup.exe"
Here is some useful links:
- How to create certificate: http://msdn.microsoft.com/en-us/library/ff699202.aspx
- pvk2pfx command keys: http://msdn.microsoft.com/en-us/library/windows/hardware/ff550672%28v=vs.85%29.aspx
- signtool command keys: http://msdn.microsoft.com/en-us/library/windows/desktop/aa387764%28v=vs.85%29.aspx
- How to sign a file: http://msdn.microsoft.com/en-us/library/windows/desktop/aa388170%28v=vs.85%29.aspx