The problem only occurs on DSA keys with a size >=2048 bits! So e.g. the following fails
openssl dsaparam 2048 > dsaparam
openssl gendsa -out keydsa.pem dsaparam
xmlsec1 sign --output doc-signed-x509.xml --privkey-pem keydsa.pem unsigned.xml
while the following works
openssl dsaparam 2047 > dsaparam
openssl gendsa -out keydsa.pem dsaparam
xmlsec1 sign --output doc-signed-x509.xml --privkey-pem keydsa.pem unsigned.xml
time for a bug report?