How to Verify a RSA-SHA512 XML Signature in .NET?

Question 1

You can verfify RSA SHA512 signatures but you'll have to implement and register the signature description by yourself.

Signature description:

public sealed class RSAPKCS1SHA512SignatureDescription : SignatureDescription
{
    public RSAPKCS1SHA512SignatureDescription()
    {
        KeyAlgorithm = typeof( RSACryptoServiceProvider ).FullName;
        DigestAlgorithm = typeof( SHA512Managed ).FullName;
        FormatterAlgorithm = typeof( RSAPKCS1SignatureFormatter ).FullName;
        DeformatterAlgorithm = typeof( RSAPKCS1SignatureDeformatter ).FullName;
    }

    public override AsymmetricSignatureDeformatter CreateDeformatter( AsymmetricAlgorithm key )
    {
        if( key == null )
        {
            throw new ArgumentNullException( "key" );
        }

        var deformatter = new RSAPKCS1SignatureDeformatter( key );
        deformatter.SetHashAlgorithm( "SHA512" );
        return deformatter;
    }

    public override AsymmetricSignatureFormatter CreateFormatter( AsymmetricAlgorithm key )
    {
        if( key == null )
        {
            throw new ArgumentNullException( "key" );
        }

        var formatter = new RSAPKCS1SignatureFormatter( key );
        formatter.SetHashAlgorithm( "SHA512" );
        return formatter;
    }
}

In your code you'll have to register this description with CryptoConfig:

const string XmlDsigRsaSha512 = "http://www.w3.org/2001/04/xmldsig-more#rsa-sha512";
CryptoConfig.AddAlgorithm( typeof( RSAPKCS1SHA512SignatureDescription ), XmlDsigRsaSha512 );

I tested it with .Net 4.0 on Windows 7 64 Bit.

Question 2

According to my research, only following signature methods are supported by the SignedXml implementation:

http://www.w3.org/2000/09/xmldsig#hmac-sha1
http://www.w3.org/2001/04/xmldsig-more#hmac-sha256
http://www.w3.org/2001/04/xmldsig-more#hmac-sha384
http://www.w3.org/2001/04/xmldsig-more#hmac-sha512
http://www.w3.org/2001/04/xmldsig-more#hmac-md5
http://www.w3.org/2001/04/xmldsig-more#hmac-ripemd160

These can be used to both sign and verify. Unfortunately, the

http://www.w3.org/2001/04/xmldsig-more#rsa-sha512

used as your signature algoritm is not supported.

Ultimately, all crypto methods go down to CryptoConfig.CreateFromName where the rsa-sha512 returns null.

Edit: I might have just found a way to make it work. Following snippet works for me:

        Dictionary<string, object> ht =
            (Dictionary<string, object>)typeof( CryptoConfig ).InvokeMember( 
            "DefaultNameHT", System.Reflection.BindingFlags.GetProperty | 
            System.Reflection.BindingFlags.Static | 
            System.Reflection.BindingFlags.NonPublic, null, typeof( CryptoConfig ), 
            null );

        var o = ht["http://www.w3.org/2000/09/xmldsig#rsa-sha1"];
        ht["http://www.w3.org/2001/04/xmldsig-more#rsa-sha512"] = o;

This should be called before you sign/validate.

This is based on the observation that the actual hash verification comes from the certificate and the algorithm name is only used as a guard. If you trick the configuration to think that the RSA-SHA512 is supported (by pointing to the same RSA-SHA1 formatter which is not used), things start to work.

Edit2: After further investigation involving consulting sources

http://www.dotnetframework.org/default.aspx/4@0/4@0/untmp/DEVDIV_TFS/Dev10/Releases/RTMRel/ndp/clr/src/ManagedLibraries/Security/System/Security/Cryptography/Xml/SignedXml@cs/1305376/SignedXml@cs

I think that the above solution will not work. What it does it only changes the signature name in the signed document, however unfortunately the signature is still computed using RSA-SHA1.

The only way to make it work would be to implement the RSA-SHA512 as KeyedHashAlgoritm as both signing and verification seem to support it with overloaded versions:

signedXml.ComputeSignature( KeyedHashAlgorithm hash );
signedXml.CheckSignature( KeyedHashAlgorithm hash );