Signing with xmlsec

Question

I can't seem to get any xmlsec1 sign call to be working; I tried several scenarios now, and the best I can come up with is the following error message:

root@sbe:/srv/jarsigner-keystore# xmlsec1 sign --privkey-pem MEDELEXIS_SWD_SRV.p12.key.pem unsignedTemplate.xml 
func=xmlSecOpenSSLDsaSha1EvpSign:file=signatures.c:line=639:obj=unknown:subj=unknown:error=11:invalid size:size(r)=32 or size(s)=32 > 20
func=xmlSecOpenSSLEvpSignatureExecute:file=signatures.c:line=491:obj=dsa-sha1:subj=EVP_SignFinal:error=4:crypto library function failed: 
func=xmlSecTransformDefaultPushBin:file=transforms.c:line=2207:obj=dsa-sha1:subj=xmlSecTransformExecute:error=1:xmlsec library function failed:final=1
func=xmlSecTransformIOBufferClose:file=transforms.c:line=2891:obj=dsa-sha1:subj=xmlSecTransformPushBin:error=1:xmlsec library function failed: 
func=xmlSecTransformC14NPushXml:file=c14n.c:line=279:obj=exc-c14n:subj=xmlOutputBufferClose:error=5:libxml2 library function failed: 
func=xmlSecTransformCtxXmlExecute:file=transforms.c:line=1236:obj=unknown:subj=xmlSecTransformPushXml:error=1:xmlsec library function failed:transform=exc-c14n
func=xmlSecDSigCtxProcessSignatureNode:file=xmldsig.c:line=614:obj=unknown:subj=xmlSecTransformCtxXmlExecute:error=1:xmlsec library function failed: 
func=xmlSecDSigCtxSign:file=xmldsig.c:line=303:obj=unknown:subj=xmlSecDSigCtxSigantureProcessNode:error=1:xmlsec library function failed: 
Error: signature failed 
Error: failed to sign file "unsignedTemplate.xml"

I don't have a clue what invalid size:size(r)=32 or size(s)=32 > 20 is all about?! I tried several combinations of keys with pkcs12, converting to pem, der where everytime the above mentioned error is the best I can get. The original certificate was created using gnomint, so I guess that should be correct!

The only thing missing, as I use xmlsec 1.2.18 is to try compile 1.2.19 with the newer openssl ... but after some hours this just raises new questions which would have to be put to stackoverflow!

Any help appreciated!

Answer 1

The problem only occurs on DSA keys with a size >=2048 bits! So e.g. the following fails

openssl dsaparam 2048 > dsaparam 
openssl gendsa -out keydsa.pem dsaparam
xmlsec1 sign --output doc-signed-x509.xml --privkey-pem keydsa.pem unsigned.xml

while the following works

openssl dsaparam 2047 > dsaparam 
openssl gendsa -out keydsa.pem dsaparam
xmlsec1 sign --output doc-signed-x509.xml --privkey-pem keydsa.pem unsigned.xml

time for a bug report?