yaronn
Difference from javax.xml.crypto.dsig and xml-crypto signature value
This is the SignedInfo canonilized
<ds:SignedInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"></ds:CanonicalizationMethod><ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"></ds:SignatureMethod><ds:Reference URI="key-info-id"><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"></ds:Transform></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"></ds:DigestMethod><ds:DigestValue>aUK42Eje/aj9EEEZn1wHoSsQMnPW0J8GpDP124YLE1w=</ds:DigestValue></ds:Reference><ds:Reference URI=""><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"></ds:Transform><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"></ds:Transform></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"></ds:DigestMethod><ds:DigestValue>3lnGNn7gACFoylCvc8RXyyg3etLlVjfuxHYcOZhTL+c=</ds:DigestValue></ds:Reference><ds:Reference><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"></ds:Transform></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"></ds:DigestMethod><ds:DigestValue>aIc0+7BXP+Ee09zng0JE21rkzgVlYaa6qMGHIBQKvTc=</ds:DigestValue></ds:Reference></ds:SignedInfo>
it generates this hash in node
QOyatmgHiGguYkBeNFUJU3b/448S7BnkNA79II01aBrsoP0AlKXAc6wAxh5B9vIRddQrMkOAiC6o8qGF2NVDd7dyNZZ5PeHyMnV88c+MbX4v68w+uVaK4nG2fFucA6Uyvl2UciohGW9bcRk3O6ImF/0tIxyuiZETEYDAd5VYTcLvWgFS2yDMFF+8FadhcFaBB4y79++YK4fCgbyUzgywfoozr+5Q7vyVxVXbhD+rb5e74v/Jg/nkS0fUmQWT2WFt9ZeSYAc/+HgMT3zlADnP6DJHgoPd/Ptmz2SnaUnolAzhFOPFL9zDG0EfWfGBJekjts/EKuq4qC6mCfwdtF/mng==
and it generates this hash in java/kotlin
XQJ8s0LlWL75DwXJ3LPZy6e8JOvkq6ccpQssQVIkr66CWGfgz2WVC0qz6s7duHhbSZxicpgYGQME
cwAfrX4rNAEpI3+NcIS7c2Mue2X3BNovfk6mBzB5bSci4bVwB2vWK0lErnjPXHAviYq2lM6YdV50
V6hiNwLNIqsLsv7k4ECkY9H0aD693K7yc1n3KT3jrEtql4uouOZbpV1LbFr+vWs0A6aMbJOm7zFz
f26gBtgjDlqRIEBgyItG+O4EBKV4DCcxPXvAS4Qx/RXiIz2wUH6EcXvdw40+VmgwgBkWHvCkCQKM
KIjEgqyv5jv8pIlPYdttYJq3Ip+vL03Sbtkb5A==
do you have an idea why java signature has #xD;\n ?
do you have an idea why java signature has #xD;\n ?
Yes. At least I have an idea. Dunno about others.
BTW. You considered this to be a bug at xml-crypto due to reporting bug issue instead of discussion post.
Whats your xml-crypto version and node version and steps to reproduce bug.
(What was your exact JDK version)
In case your JDK version was some of those affected by following:
- https://bugs.openjdk.org/browse/JDK-8264194
- https://github.com/node-saml/node-saml/issues/242#issuecomment-1374056961
- also follow link from aforementioned comment to keycloak side issue and read through comments
Here is a Java implementation https://github.com/aws-samples/pix-proxy-samples/blob/fa20042d19d4c0b78f6898f3053b4cd0729938d0/proxy/core/src/main/java/com/amazon/aws/pix/core/xml/Iso20022XmlSigner.java#L25
I suggested https://github.com/node-saml/xml-crypto/issues/486#issuecomment-2637904015 as answer to your question's #xD; part.
About splitting every 76th character (the \n part of your question)...dunno if I read spec correctly but I get a feeling that oneliner and splitted at every 76th character are both ok.
https://www.w3.org/TR/2002/REC-xmldsig-core-20020212/#sec-SignatureValue
https://www.w3.org/TR/xmlschema-2/#base64Binary
Quick internet search provided following thread regarding xmlsig's base64 stuff: https://lists.w3.org/Archives/Public/w3c-ietf-xmldsig/2001AprJun/thread.html#msg183
There are also other threads about same stuff (with same subject) at that mailing list with emails from xmlsig spec contributors.
About link you provided to pix-proxy-samples...it does not reveal anything about version of implementation of javax.xml.crypto.dsig.*. Output of java -version of the env that you used to run that example would have. Eitherway...are you saying xml-crypto has a bug or aren't you? If some java sw emits 
s due whatever reason are those handled by C14N correctly? Provide testcase which pin points actual issue with content that one can feed to whatever xmlsig tool (like xmlsec1) to gather "second opinions" (i.e. provide some signed xml instead of random snippet). You can use e.g. xmlsec1 also by yourself to check whether it validates some signed xml correctly and if it does but xml-crypto does not then there might be issue at xml-crypto side (in that case provide also exact commands how you validated document with some other tool (e.g. with xmlsec1). Consider also checking producer side (in this case pix-proxy-samples) and consider reading speculations from earlier indirectly referenced keycloak issue.
The Java implementation passes our internal tool validation but node xml-crypto does not
We don't have access to the internal tool validation of the signature
I just wanna produce the same signature value both in Java and node
Im trying to understand why they differ and how to make both the same
- Give example of signed xml document (along with certificate of key that was used to sign) that passes shell script which uses
xmlsec1 --verify ...(construct shell script so that it has everything needed to verify signature. I.e. certificate and signed xml document which shell script passes to xmlsec1) and example of JS code which uses xml-crypto (and xml-crypto fails to validate that same xml document). - and/or give example of JS which uses xml-crypto to sign some XML so that xmlsec1 AND your internal tool fails to validate it (if xmlsec1 validates it correctly but your internal tool fails then you should debug your internal tool)
Having such material gives something to work with. Without such material continuation of this discussion is pointless.
Here is example how xmlsec1 was used at one case as a reference implementation: https://github.com/node-saml/xml-crypto/issues/212#issuecomment-1949310736
Note: that case was related to signing (and more specifically to find out lack of implicit C14N during signing and how adding it explicitly lined digest values and thus signatures between different tools).
Sidenote: if you would have posted full signed document (signed by xml-crypto) one could see whether there is chance that your root cause is same.
Here is the spec https://www.bcb.gov.br/content/estabilidadefinanceira/cedsfn/Manual_de_Seguranca_PIX.pdf
It is in Portuguese (Brazil), you could easily translate it
It has some specific references style
You can also check some examples in the unit tests of the aws repo
input xml https://github.com/aws-samples/pix-proxy-samples/blob/fa20042/proxy/core/src/test/resources/xml/pacs.008_CONTA_1_msg.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<Envelope xmlns="https://www.bcb.gov.br/pi/pacs.008/1.4">
<AppHdr>
<Fr>
<FIId>
<FinInstnId>
<Othr>
<Id>00038166</Id>
</Othr>
</FinInstnId>
</FIId>
</Fr>
<To>
<FIId>
<FinInstnId>
<Othr>
<Id>99999010</Id>
</Othr>
</FinInstnId>
</FIId>
</To>
<BizMsgIdr>M0003816612345678901234567890123</BizMsgIdr>
<MsgDefIdr>pacs.008.spi.1.4</MsgDefIdr>
<CreDt>2020-01-01T08:30:12.000Z</CreDt>
<Sgntr/>
</AppHdr>
<Document>
<FIToFICstmrCdtTrf>
<GrpHdr>
<MsgId>M0003816612345678901234567890123</MsgId>
<CreDtTm>2020-01-01T08:30:12.000Z</CreDtTm>
<NbOfTxs>1</NbOfTxs>
<SttlmInf>
<SttlmMtd>CLRG</SttlmMtd>
</SttlmInf>
<PmtTpInf>
<InstrPrty>HIGH</InstrPrty>
</PmtTpInf>
</GrpHdr>
<CdtTrfTxInf>
<PmtId>
<EndToEndId>E9999901012341234123412345678900</EndToEndId>
<TxId>90000</TxId>
</PmtId>
<IntrBkSttlmAmt Ccy="BRL">1000.00</IntrBkSttlmAmt>
<AccptncDtTm>2020-01-01T08:30:00.000Z</AccptncDtTm>
<ChrgBr>SLEV</ChrgBr>
<Dbtr>
<Nm>Fulano da Silva</Nm>
<Id>
<PrvtId>
<Othr>
<Id>70000000000</Id>
</Othr>
</PrvtId>
</Id>
</Dbtr>
<DbtrAcct>
<Id>
<Othr>
<Id>500000</Id>
<Issr>3000</Issr>
</Othr>
</Id>
<Tp>
<Cd>CACC</Cd>
</Tp>
</DbtrAcct>
<DbtrAgt>
<FinInstnId>
<ClrSysMmbId>
<MmbId>10000000</MmbId>
</ClrSysMmbId>
</FinInstnId>
</DbtrAgt>
<CdtrAgt>
<FinInstnId>
<ClrSysMmbId>
<MmbId>20000000</MmbId>
</ClrSysMmbId>
</FinInstnId>
</CdtrAgt>
<Cdtr>
<Id>
<PrvtId>
<Othr>
<Id>80000000000</Id>
</Othr>
</PrvtId>
</Id>
</Cdtr>
<CdtrAcct>
<Id>
<Othr>
<Id>600000</Id>
<Issr>4000</Issr>
</Othr>
</Id>
<Tp>
<Cd>SVGS</Cd>
</Tp>
</CdtrAcct>
<RmtInf>
<Ustrd>Campo livre [0]</Ustrd>
</RmtInf>
</CdtTrfTxInf>
</FIToFICstmrCdtTrf>
</Document>
</Envelope>
signed
<?xml version="1.0" encoding="UTF-8" standalone="no"?><Envelope xmlns="https://www.bcb.gov.br/pi/pacs.008/1.4">
<AppHdr>
<Fr>
<FIId>
<FinInstnId>
<Othr>
<Id>00038166</Id>
</Othr>
</FinInstnId>
</FIId>
</Fr>
<To>
<FIId>
<FinInstnId>
<Othr>
<Id>99999010</Id>
</Othr>
</FinInstnId>
</FIId>
</To>
<BizMsgIdr>M0003816612345678901234567890123</BizMsgIdr>
<MsgDefIdr>pacs.008.spi.1.4</MsgDefIdr>
<CreDt>2020-01-01T08:30:12.000Z</CreDt>
<Sgntr><ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:SignedInfo><ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/><ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/><ds:Reference URI="#9115a29c-1246-4de2-8d33-d3c8549efac3"><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/><ds:DigestValue>Kmg8b7mLCJtFj1co5CvLBusbRQcpq2rurd87Fy1VmMA=</ds:DigestValue></ds:Reference><ds:Reference URI=""><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/><ds:DigestValue>0mtmawoDDPLwAAAkSYdy0JmPf5oS55rHZjsXOASzY4c=</ds:DigestValue></ds:Reference><ds:Reference><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/><ds:DigestValue>UP6Uy9SYzwVsO+djbyusrydi2/If22U4bEHZt/NJhH0=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue>IhddKqaN3zNXNN30h6+7RYL6tdAQZmf0UqPnrBwlL5K/cjS6J8ph3DVGODQDWZHNHXqReOiHaAzT
svR5SnBYek2/Z0rmzs1zuf3pYOKb5n2K/EnG6uLot/lWILd9oOYBJcN5eYRNF70sKcEy5pE0TuEJ
y1yh0EQKWSOACuESLNH+FH71sFpBXJSgOSj6FVQ0utZc1A4YdvPZZCsayOmSVHqwBgRSGHG05k4b
C7rX5xhXT4ezMEqk4XXTVZD42j6LFcB09Hvq5RNOqrspRlguztOwl23M6iAtsbmCfgDbG2w0B8HQ
yFw8maLMVGbfz2WONs6gRasdaEKVnEe1rhnKwg==</ds:SignatureValue><ds:KeyInfo Id="9115a29c-1246-4de2-8d33-d3c8549efac3"><ds:X509Data><ds:X509IssuerSerial><ds:X509IssuerName>CN=client.pix.aws.com,OU=PIX,O=AWS,L=Sao Paulo,ST=SP,C=BR</ds:X509IssuerName><ds:X509SerialNumber>2004099543</ds:X509SerialNumber></ds:X509IssuerSerial></ds:X509Data></ds:KeyInfo></ds:Signature></Sgntr></AppHdr>
<Document>
<FIToFICstmrCdtTrf>
<GrpHdr>
<MsgId>M0003816612345678901234567890123</MsgId>
<CreDtTm>2020-01-01T08:30:12.000Z</CreDtTm>
<NbOfTxs>1</NbOfTxs>
<SttlmInf>
<SttlmMtd>CLRG</SttlmMtd>
</SttlmInf>
<PmtTpInf>
<InstrPrty>HIGH</InstrPrty>
</PmtTpInf>
</GrpHdr>
<CdtTrfTxInf>
<PmtId>
<EndToEndId>E9999901012341234123412345678900</EndToEndId>
<TxId>90000</TxId>
</PmtId>
<IntrBkSttlmAmt Ccy="BRL">1000.00</IntrBkSttlmAmt>
<AccptncDtTm>2020-01-01T08:30:00.000Z</AccptncDtTm>
<ChrgBr>SLEV</ChrgBr>
<Dbtr>
<Nm>Fulano da Silva</Nm>
<Id>
<PrvtId>
<Othr>
<Id>70000000000</Id>
</Othr>
</PrvtId>
</Id>
</Dbtr>
<DbtrAcct>
<Id>
<Othr>
<Id>500000</Id>
<Issr>3000</Issr>
</Othr>
</Id>
<Tp>
<Cd>CACC</Cd>
</Tp>
</DbtrAcct>
<DbtrAgt>
<FinInstnId>
<ClrSysMmbId>
<MmbId>10000000</MmbId>
</ClrSysMmbId>
</FinInstnId>
</DbtrAgt>
<CdtrAgt>
<FinInstnId>
<ClrSysMmbId>
<MmbId>20000000</MmbId>
</ClrSysMmbId>
</FinInstnId>
</CdtrAgt>
<Cdtr>
<Id>
<PrvtId>
<Othr>
<Id>80000000000</Id>
</Othr>
</PrvtId>
</Id>
</Cdtr>
<CdtrAcct>
<Id>
<Othr>
<Id>600000</Id>
<Issr>4000</Issr>
</Othr>
</Id>
<Tp>
<Cd>SVGS</Cd>
</Tp>
</CdtrAcct>
<RmtInf>
<Ustrd>Campo livre [0]</Ustrd>
</RmtInf>
</CdtTrfTxInf>
</FIToFICstmrCdtTrf>
</Document>
</Envelope>
Now....how is that related to xml-crypto?
Are you saying that xml-crypto fails to verify signature? Does xmlsec1 verify that signature?
Or
Are you saying that if xml-crypto would be used to sign that document then output of xml-crypto cannot be verified by xmlsec1? Provide JS file which signs that document with xml-crypto so that we can see how you tried to do that and output also. Also keep in mind that xml-crypto is unable to sign keyinfo (as discussed at https://github.com/node-saml/xml-crypto/issues/481 ) so you have to modify your example (which is signing keyinfo also) accordingly (I gather that you are debugging another aspect).
And
where is the certificate / public key which can be used for signature verification.
It is all in the aws repo
Good. Then there chance that you hand out direct link. No-one else except youself is getting any money to run around to gather material or debugging this on behalf of you.
btw. I copy pasted your example signed xml to https://tools.chilkat.io/xmlDsigVerify.cshtml
It said:
Signature is Invalid
Number of Reference Digests = 3
Reference 1 digest is valid.
Reference 2 digest is invalid because the computed digest differs from the digest in the XML.
Reference 3 digest is invalid because the computed digest differs from the digest in the XML.
Note: there is no way chilkat could say that signature is valid because cert is not available.
But now you have another track which is: explain yourself why unrelated (from xml-crypto pov) tool is unable to calculate same digest values as your reference implementation. Consider consulting also xmlsec1 tool's opinion. Lets continue after you have sorted things out between your reference implementation and xmlsec1 and chilkat. And keep in mind that no matter what xml-crypto is currently unable to sign keyinfo.
I could made xml-crypto to calculate the same digest
I have a custom implementation to sign KeyInfo, not generic enough for this package yet
The digest value are correct and the same as Java implementation
However the signature value has this difference
- Show full output (signed xml document) of your xml-crypto fork and your java sw. Use same input (private key and input xml) for both cases.
- I am especially interest to see what transformations you have added for references when signing with xml-crypto (keep in mind that it does not apply implicitly that one C14N transformation - in case there wasn't any C14N algorithm at transformation list - during signing as other tools apply which means that due to that issue actual values of signature shall be different unless you add it explicitly ... yes it would pop up to list of transformations but it does not matter it just means that you made it explicit)
- If actual values of signature matches but java implementation split signature to multiple lines and adds that
#xD;(while xml-crypto writes oneliner) then tell us what xmlsec1 thinks about java and xml-crypto versions when you use it to verify signatures.
- tell us what xmlsec1 says about your java and xml-crypto versions of signed xml even if they have different signature (from actual value point of view or from "layout" point of view) but otherwise 1:1 content. Is either of your implementations generating correct signature from xmlsec1 point of view.
Adding more context to this issue for future generations who might land here via some search engine.
Based on content of this comment: https://github.com/node-saml/xml-crypto/issues/486#issuecomment-2638329029
This issue seems to be at the end of the day "same" as:
- https://github.com/node-saml/xml-crypto/issues/204
or at least very close to that in terms of how signature should be constructed (if we think how this issue might be related to xml-crypto).
Author of https://github.com/node-saml/xml-crypto/issues/204 was kind enought to describe expected signature content (without having to use any translator to translate portuguese document):
Where URI='' is the signature from the AppHdr without the Signature element, URI='uuid' is the signature from the KeyInfo(A x509 certificate) and ds:Reference is the signature from the Document element.
Regarding Reference which URI attribute is omitted spec https://www.w3.org/TR/2008/REC-xmldsig-core-20080610/#sec-URI says:
If the
URIattribute is omitted altogether, the receiving application is expected to know the identity of the object. For example, a lightweight data protocol might omit this attribute given the identity of the object is part of the application context. This attribute may be omitted from at most oneReferencein any particularSignedInfo, orManifest.
So (IMHO) it seems that at the end of the day one has to construct signature without the help of xml-crypto because at least I can't figure out how xml-crypto would be made aware which part of the document should be handled when reference without URI is added.
This second opinion backs aforementioned interpretation: https://stackoverflow.com/questions/15522098/java-xmldsig-reference-with-no-uri/18526152#18526152
About those extra xml entities (which started this whole discussion)...its something java stack related.
I have a custom implementation to sign KeyInfo, not generic enough for this package yet
@sibelius , are you interested in sharing what you've developed, even if it isn't generic? Perhaps someone else would be interested in making it more generic.
So I take it that you're abandoning this code and won't be sharing what you're written? If so, we'll just close this issue. Please LMK.