signxml
signxml copied to clipboard
XML-Security
Incorrect signature in 2.10.0
A test in my project revealed that signxml 2.10.0 generates invalid signatures, while 2.9.0 generates correct signatures. The cause is the new excise_empty_xmlns_declarations parameter, which is appropriately set when validating a signature, but it also needs to be set when generating a signature.
Thank you for the report. Can you please provide a complete reproduction, including the document that was signed, the configuration that you used, and the counterparty implementation that you used to judge whether the signature was "correct" vs. "incorrect"?
After looking at it in more depth, I think you are correct that the option as introduced is applied inconsistently (and there is a test coverage gap for this use case).
I have pushed a change that makes the option apply uniformly. Can you please test with this change and verify that it works for you?
@kislyuk, it would be ideal to keep the same behavior before creating the excise_empty_xmlns_declarations parameter, that is, by default, use excise_empty_xmlns_declarations = True here, this would work on versions <= 2.9.0
@leogregianin this behavior is not part of any standard that I can identify, and it was implemented via a fragile hack. Do you have a counterparty application or other peer implementation of XML signature that depends on it?
(See discussion in #193 for details - while there is some discussion of this behavior in the xml-c14n standard, the discussion does not include clear guidance on when to apply excision and the lxml c14n implementation does not perform it)
Signing XML files with method assinar from the PyNFe project, which is a library for generating, signing and send tax documents in Brazil.
Until 2.9 version we always stripped the xmlns="" from the signature because _c14n method from signxml exclusive variable would always be false.
In 2.10 version, if is not explicitly included parameter excise_empty_xmlns_declarations = True, xmlns="" is not excluded from the signature, resulting in incorrect validation.
The small part of the signature generated by signxml method def _c14n(self, nodes, algorithm, inclusive_ns_prefixes= None, excise_empty_xmlns_declarations=False): returns this:
<SignedInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
<CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/>
<SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
<Reference URI="#NFe41210199999999000199550010000001111904477123">
<Transforms xmlns="">
<Transform xmlns="" Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
<Transform xmlns="" Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/>
</Transforms>
<DigestMethod xmlns="" Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
<DigestValue xmlns="">koJdFRsHM7PqULSY6vgHzDdCzN0=</DigestValue>
</Reference>
</SignedInfo>
I don't know exactly what we can do.
To maintain compatibility between versions 2.9 and 2.10 would have to set excise_empty_xmlns_declarations = True with default class variable in signxml.
Or require projects that use version >=2.10 and set excise_empty_xmlns_declarations = True.
In theory, the xmlns=""attributes shouldn't even appear in the canonical XML if the tags inside SignedInfo all belong to the same xmldsig namespace. Perhaps signxml isn't (or wasn't) using XML namespaces correctly for SignedInfo.
How do I sign the XML:
signer = XMLSigner(
method=signxml.methods.enveloped,
signature_algorithm="rsa-sha1",
digest_algorithm='sha1',
c14n_algorithm='http://www.w3.org/TR/2001/REC-xml-c14n-20010315'
)
lxml etree.tostring includes xmlns="" when the exclusive parameter is False.
This behavior happens inside the concat c14n += etree.tostring in _c14n method of signxml.
In this case, the exclusive parameter should be True?
@leogregianin if you wish to use exclusive XML canonicalization, you have to specify that using the corresponding value in c14n_algorithm.
It looks like this issue only manifests when you override the default namespace (i.e. modify XMLSigner.namespaces to include {"": "http://www.w3.org/2000/09/xmldsig#"}):
>./test/test.py -vvv TestSignXML.test_changing_signature_namespace_prefix_to_default
test_changing_signature_namespace_prefix_to_default (__main__.TestSignXML) ... =======BEGIN C14N=========
<data>
<country name="Liechtenstein">
<rank>1</rank>
<year>2008</year>
<gdppc>141100</gdppc>
<neighbor direction="E" name="Austria"></neighbor>
<neighbor direction="W" name="Switzerland"></neighbor>
</country>
<country name="Singapore">
<rank>4</rank>
<year>2011</year>
<gdppc>59900</gdppc>
<neighbor direction="N" name="Malaysia"></neighbor>
</country>
<country name="Panama">
<rank>68</rank>
<year>2011</year>
<gdppc>13600</gdppc>
<neighbor direction="W" name="Costa Rica"></neighbor>
<neighbor direction="E" name="Colombia"></neighbor>
</country>
</data>
======= END C14N=========
=======BEGIN C14N=========
<SignedInfo xmlns="http://www.w3.org/2000/09/xmldsig#"><CanonicalizationMethod Algorithm="http://www.w3.org/2006/12/xmlc14n11#"></CanonicalizationMethod><SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"></SignatureMethod><Reference URI=""><Transforms xmlns=""><Transform xmlns="" Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"></Transform><Transform xmlns="" Algorithm="http://www.w3.org/2006/12/xmlc14n11#"></Transform></Transforms><DigestMethod xmlns="" Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"></DigestMethod><DigestValue xmlns="">phz8KnRySaYxGRtVaEG1/XPY+8JRkEcl0HIPl8ySSfs=</DigestValue></Reference></SignedInfo>
======= END C14N=========
ok
----------------------------------------------------------------------
Ran 1 test in 0.162s
OK
In other words, I think the issue is that the custom namespace map is being used incorrectly in the canonicalization process.
I don't think this is a bug.
SignXML relies on lxml for canonicalization, so even if this was a bug, it could only be a misconfiguration of lxml by SignXML, but I don't believe this to be true.
Quoting https://www.w3.org/TR/xmldsig-core1/#sec-CoreGeneration:
3.1.2 Signature Generation Create SignedInfo element with SignatureMethod, CanonicalizationMethod and Reference(s). Canonicalize and then calculate the SignatureValue over SignedInfo based on algorithms specified in SignedInfo.
XML Signature uses XML Canonicalization 1.1 (or 1.0, which is not meaningfully different for this issue). Quoting https://www.w3.org/TR/xml-c14n/#ProcessingModel:
2.3 Processing Model The XPath node-set is converted into an octet stream, the canonical form, by generating the representative UCS characters for each node in the node-set in ascending document order, then encoding the result in UTF-8 (without a leading byte order mark). No node is processed more than once. Note that processing an element node E includes the processing of all members of the node-set for which E is an ancestor. Therefore, directly after the representative text for E is generated, E and all nodes for which E is an ancestor are removed from the node-set (or some logically equivalent operation occurs such that the node-set's next node in document order has not been processed). Note, however, that an element node is not removed from the node-set until after its children are processed.
The result of processing a node depends on its type and on whether or not it is in the node-set. If a node is not in the node-set, then no text is generated for the node except for the result of processing its namespace and attribute axes (elements only) and its children (elements and the root node). If the node is in the node-set, then text is generated to represent the node in the canonical form in addition to the text generated by processing the node's namespace and attribute axes and child nodes.
NOTE: The node-set is treated as a set of nodes, not a list of subtrees. To canonicalize an element including its namespaces, attributes, and content, the node-set must actually contain all of the nodes corresponding to these parts of the document, not just the element node.
The text generated for a node is dependent on the node type and given in the following list:
Root Node- The root node is the parent of the top-level document element. The result of processing each of its child nodes that is in the node-set in document order. The root node does not generate a byte order mark, XML declaration, nor anything from within the document type declaration. Element Nodes- If the element is not in the node-set, then the result is obtained by processing the namespace axis, then the attribute axis, then processing the child nodes of the element that are in the node-set (in document order). If the element is in the node-set, then the result is an open angle bracket (<), the element QName, the result of processing the namespace axis, the result of processing the attribute axis, a close angle bracket (>), the result of processing the child nodes of the element that are in the node-set (in document order), an open angle bracket, a forward slash (/), the element QName, and a close angle bracket. Namespace Axis- Consider a list L containing only namespace nodes in the axis and in the node-set in lexicographic order (ascending). To begin processing L, if the first node is not the default namespace node (a node with no namespace URI and no local name), then generate a space followed by xmlns="" if and only if the following conditions are met:
the element E that owns the axis is in the node-set The nearest ancestor element of E in the node-set has a default namespace node in the node-set (default namespace nodes always have non-empty values in XPath) The latter condition eliminates unnecessary occurrences of xmlns="" in the canonical form since an element only receives an xmlns="" if its default namespace is empty and if it has an immediate parent in the canonical form that has a non-empty default namespace. To finish processing L, simply process every namespace node in L, except omit namespace node with local name xml, which defines the xml prefix, if its string value is http://www.w3.org/XML/1998/namespace.
Attribute Axis- In lexicographic order (ascending), process each node that is in the element's attribute axis and in the node-set.
I've highlighted the relevant parts in bold. Essentially, if you instruct SignXML to set the namespace prefix for the http://www.w3.org/2000/09/xmldsig# namespace in your XML signature to the empty (default) namespace prefix instead of the customary ds, then for any canonicalization done as part of the signature (in particular SignedInfo), the immediate child elements of SignedInfo will contain no prefix and no xmlns attribute, and their child elements (non-immediate/skip-level elements) will contain xmlns="" as an attribute.
This lxml behavior is consistent with the standard and there is nothing to fix. SignXML before 2.10.0 erroneously removed these attributes and we will not be going back to that behavior. The excise_empty_xmlns_declarations will be marked as deprecated in v3.0.0 and removed in a future release.
@hathawsh for future bug reports, please provide references and/or examples instead of just claiming that the behavior is "incorrect".
Recommended workaround if you encounter this issue
Do not set the empty (default) namespace prefix (XMLSigner.namespaces = ...) when generating the signature.
What if my counterparty requires the use of unprefixed elements in the XML signature?
XML signature processors should be namespace aware and should support arbitrary namespace prefix remapping. SignXML does not support interoperability with peer applications that have bugs in both their namespace prefix assumptions and canonicalization methods.
@kislyuk , thanks for looking into this. It takes some time for me to isolate and sanitize a test case.
This test only works with the branch of signxml I created at https://github.com/OpenPaymentNetwork/signxml . The company I'm working with accepts the signatures generated by our branch; it does not accept the signatures generated by signxml 2.10. That's why I believe our branch is correct and signxml 2.10 is not. I am certainly open to the possibility that we're calling signxml in some incorrect way.
The private key I included is used only for this test case and nothing else, so there's no risk in sharing it.
import signxml
from cryptography.hazmat.primitives.serialization import load_pem_private_key
from lxml.etree import Element, ElementTree, SubElement, fromstring, tostring
from OpenSSL.crypto import FILETYPE_PEM, load_certificate
signable = """\
<Message xmlns="urn:someco" xmlns:head="urn:iso:std:iso:20022:tech:xsd:head.001.001.01">\
<AppHdr>\
<head:Fr>\
<head:FIId>\
<head:FinInstnId>\
<head:ClrSysMmbId>\
<head:MmbId>11234567890</head:MmbId>\
</head:ClrSysMmbId>\
</head:FinInstnId>\
</head:FIId>\
</head:Fr>\
<head:To>\
<head:FIId>\
<head:FinInstnId>\
<head:ClrSysMmbId>\
<head:MmbId>98765432100</head:MmbId>\
</head:ClrSysMmbId>\
</head:FinInstnId>\
</head:FIId>\
</head:To>\
<head:BizMsgIdr>B2022010201234567890BTST00000087654</head:BizMsgIdr>\
<head:MsgDefIdr>admn.005.001.01</head:MsgDefIdr>\
<head:CreDt>2022-01-02T03:04:05</head:CreDt>\
<head:Sgntr/>\
</AppHdr>\
</Message>\
"""
expected_output = """\
<Message xmlns="urn:someco" xmlns:head="urn:iso:std:iso:20022:tech:xsd:head.001.001.01">
<AppHdr>
<head:Fr>
<head:FIId>
<head:FinInstnId>
<head:ClrSysMmbId>
<head:MmbId>11234567890</head:MmbId>
</head:ClrSysMmbId>
</head:FinInstnId>
</head:FIId>
</head:Fr>
<head:To>
<head:FIId>
<head:FinInstnId>
<head:ClrSysMmbId>
<head:MmbId>98765432100</head:MmbId>
</head:ClrSysMmbId>
</head:FinInstnId>
</head:FIId>
</head:To>
<head:BizMsgIdr>B2022010201234567890BTST00000087654</head:BizMsgIdr>
<head:MsgDefIdr>admn.005.001.01</head:MsgDefIdr>
<head:CreDt>2022-01-02T03:04:05</head:CreDt>
<head:Sgntr>
<Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
<SignedInfo>
<CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/>
<SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
<Reference URI="">
<Transforms>
<Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
<Transform Algorithm="http://www.w3.org/2006/12/xml-c14n11"/>
</Transforms>
<DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
<DigestValue>PP97CV3msqL3yEmD5fIVoJ9QOhVbacGUBGUq1kMENd4=</DigestValue>
</Reference>
</SignedInfo>
<SignatureValue>VbEXHdNUv5KdJp8Wljzc2wxWM3OwqsfxWDA8L0NyZxfzh/Z7JLRzUiZ8p4ge1NB284aOUNBoSinEmLWoX2PEfbzptTgYm6Z3HGBFetLuhoIvwVzOywe4OkwEDi47ku4fWSL7D7tLwe8gGDyUQSf499I/VJnksPTE8EPrCphr+Ytv5D7jSurqobc9FhHBAhftcd40gEAruQzXsuem9b2mrr+oivqgIjGibNwbuc78xdIGbyuIQ1ybSRZj44tf1lzJ+r4/Je7q0oWoHeO2doeIMXkt59fY7n9CJLLbRGqkq2179OTwrktrvffiz8FOWOxwj/y18oDjuLL9tzPDBEsjEw==</SignatureValue>
<KeyInfo>
<X509Data>
<X509SubjectName>CN=test.localdomain,C=US</X509SubjectName>
<X509IssuerSerial>
<X509IssuerName>CN=test.localdomain,C=US</X509IssuerName>
<X509SerialNumber>508916181263576492682568652896711058344409241242</X509SerialNumber>
</X509IssuerSerial>
</X509Data>
</KeyInfo>
</Signature>
</head:Sgntr>
</AppHdr>
</Message>
"""
testcert = b"""\
-----BEGIN CERTIFICATE-----
MIIDMzCCAhugAwIBAgIUWSSXQEsgbnlB9LuLL8aAqyaXFpowDQYJKoZIhvcNAQEL
BQAwKDELMAkGA1UEBhMCVVMxGTAXBgNVBAMMEHRlc3QubG9jYWxkb21haW4wIBcN
MjIxMTE0MTYzMzExWhgPMjEyMjExMTQxNjMzMTFaMCgxCzAJBgNVBAYTAlVTMRkw
FwYDVQQDDBB0ZXN0LmxvY2FsZG9tYWluMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
MIIBCgKCAQEAxRJ0+uSUxVhrJSgFd/KNAO00lHlTt28gCxYPtZ1lh7Y34dA3RfEQ
KBeNbZftLwSjaekFyH7q2FsVwNTAuFuCVNSd8s2qLA7lg9YIAqYH72tom0hCvvjL
XdQmZoBs34pjFlEXIELpUJb+Zej/pFKXCJXO44rVKPmWSJ7QxcxcuC+60xnKr0Ep
RB/cFuYhEt6x8G5K1zgfO+IDkXCZilpzlUMfdXmSQFM01JAR9MXkAL+M4apYOQI2
q0fC6whLCaExUmek890uBHgUTTFUE6EqZlCSjm3JU4rjqRgyq7d8wIfl81POxjzn
+WFheP0LYv0tM0HHI6UGDMnsBw97QvhgwwIDAQABo1MwUTAdBgNVHQ4EFgQUAhHg
NYhCQ+sxMQLNwySM+1w3H5IwHwYDVR0jBBgwFoAUAhHgNYhCQ+sxMQLNwySM+1w3
H5IwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAR9vo+FRn4qpp
0GqH9tLAM3PdOo6twHkIHF+1TVmdsKbYN55x23JFOin0aJfp3wNgj23JQbkLGFGN
UHEL9QvLV0CdEAcHPJz6l9uJJYeOsW1XmSe6EwHHSXgkIj+btB/oL8wuPJJSTc8e
+3XiSvOdX4brvELwxIqNKwdpUvv8Bip0lMyc9BGTMfIBSBhuAYrw/vef17SZL3VW
6/MbU/CnMtwvVKFFVFZUt3AHLd7t2mW69QUfypxm51/HwRZtiM91GNplzFyKxbD3
QQ4qGZnb9uJwr6KVojHpQ7aVU2u/SQHlNQecEIFBPE1iJ/2eghGfdUHoLyXb/x9k
aSmD07X14Q==
-----END CERTIFICATE-----
"""
testkey = b"""\
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDFEnT65JTFWGsl
KAV38o0A7TSUeVO3byALFg+1nWWHtjfh0DdF8RAoF41tl+0vBKNp6QXIfurYWxXA
1MC4W4JU1J3yzaosDuWD1ggCpgfva2ibSEK++Mtd1CZmgGzfimMWURcgQulQlv5l
6P+kUpcIlc7jitUo+ZZIntDFzFy4L7rTGcqvQSlEH9wW5iES3rHwbkrXOB874gOR
cJmKWnOVQx91eZJAUzTUkBH0xeQAv4zhqlg5AjarR8LrCEsJoTFSZ6Tz3S4EeBRN
MVQToSpmUJKObclTiuOpGDKrt3zAh+XzU87GPOf5YWF4/Qti/S0zQccjpQYMyewH
D3tC+GDDAgMBAAECggEAS1BwcGnnCTxvRjXzURHbHV2J2Kw9eD75ygLWYhwS7ziO
RrhiR1KHUCsgG6ASPwPQx+sMEMGgUqsMtzMXq90dA/p1mpNK06elwyqKyShWpAMy
aRXGnGFMp0Eqo8W01gF3ONIoumWX8PqjoqLXAB+oJNnLxV4LdKtd6cb4Fn+xd5nq
G4FDZxrdEb+Kzz36egCC9O9AJdxh2H+Z0hq8WBVFPedcwxK7OBSP2H/9LmPV+Ulp
+ol2xvtyIixOuAZ91dMi/XOJVQnsCjuqRfd+USB1OR58/0LR3zt37F5bRqddM71R
dwL8qrRoFlkuWJPBY68ACZdSI60XnYKfmPQ0jyNdIQKBgQDuQ44duRjtq8jWzH6o
jAXUoyHHBtoY41SUCJ/fbbP+N/3fyXKpHkH/hpL3iY/CMz0ddflleFps8rfm0gyM
10uIzFRxmjqFXdumsB/XVD5QDYpRSBGPgAoQOWM95UDdAZ+0y6rwfIFlj5mQabPF
bbneD+TILnMNYiGVv5n0hxk8OQKBgQDTve+Re7RdEs3TfCckOPjTmPQ7zfMdNkF9
LCotpNdiTM7mpBSAeu65AQuUZQcHcOlpOiUBtjo/Cy+Bc+DXmnAPwHP8vh4JuunR
YEukmpo1EwxkQRvCa/LnL3Qd1WuwVWtBm5A31C2CLj1FgIP78SEsVtLEa8oR6CVU
tZrMOWi82wKBgQDInECwbgS7OIRlttB+AUd3fdMWzIIdqmqwKYLSkH+YcHRlVCwi
kMNo7loX6c8M6C9w+r/925PnX4XP5tOCUlPa3o/LBtx3Jkum0Ww91USSCpSHU3LV
rCzM9ETG/JvceV2K0GEhEvOwG3Rkz1r5xCVW/3LVyaY9gn5co3JxHL72oQKBgBU/
3Xf7MymYBoHv1rnC4e3STshlrb1DwaS/Nuhnv5nE3Yq18rzGGYkFetUEooTzpukB
hc87K3NCOx1BtNHOVOqvxLQbnEYbtPVnNZAqV8l1xOuWwSNs0+6xi3SnA2yp1d66
rul7aKnE3C9Ka3RwSYT6naJKsvfDFWP/6a4PINy7AoGBAMv6NHe4wXs7koIZXtWp
SZ2km6TS0qTpHNesSRodn7gldqMgqJPbZCNNkW3HkHQdp4kszufpaFagrrEmV3Kk
IoKL4RO2+FBQgryGgYgRohDtuXASoct6ZsxMlQk7IgEtzmQnBQnU2JWo6xjf2Xtp
877zDFys+ozVSNmsMqOF5Ad/
-----END PRIVATE KEY-----
"""
def make_key_info_elem(cert):
elem = Element("{http://www.w3.org/2000/09/xmldsig#}KeyInfo")
data = SubElement(elem, "{http://www.w3.org/2000/09/xmldsig#}X509Data")
subj = SubElement(data, "{http://www.w3.org/2000/09/xmldsig#}X509SubjectName")
subj.text = "CN=test.localdomain,C=US"
iserial = SubElement(data, "{http://www.w3.org/2000/09/xmldsig#}X509IssuerSerial")
iname = SubElement(iserial, "{http://www.w3.org/2000/09/xmldsig#}X509IssuerName")
iname.text = "CN=test.localdomain,C=US"
inum = SubElement(iserial, "{http://www.w3.org/2000/09/xmldsig#}X509SerialNumber")
inum.text = f"{cert.get_serial_number()}"
return elem
def test_sig():
nsmap_xmldsig = {None: "http://www.w3.org/2000/09/xmldsig#"}
signable_tree = ElementTree(fromstring(signable))
sig_envelope = signable_tree.find(
"//{urn:iso:std:iso:20022:tech:xsd:head.001.001.01}Sgntr"
)
assert sig_envelope is not None
SubElement(
sig_envelope,
"{http://www.w3.org/2000/09/xmldsig#}Signature",
attrib={"Id": "placeholder"},
nsmap=nsmap_xmldsig,
)
ssl_cert = load_certificate(FILETYPE_PEM, testcert)
ssl_key = load_pem_private_key(testkey, password=None)
key_info_elem = make_key_info_elem(ssl_cert)
signer = signxml.XMLSigner(
signature_algorithm="rsa-sha256",
c14n_algorithm="http://www.w3.org/2006/12/xml-c14n11",
signed_info_c14n_algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315",
digest_algorithm="sha256",
)
signer.namespaces = nsmap_xmldsig
signed_root = signer.sign(
signable_tree,
key=ssl_key,
cert=ssl_cert,
key_info=key_info_elem,
)
signed_output = tostring(signed_root, encoding="unicode")
print(signed_output)
signed_elem = fromstring(signed_output)
pretty = tostring(signed_elem, pretty_print=True, encoding="unicode")
print(pretty)
assert pretty == expected_output, "mismatched"
if __name__ == "__main__":
test_sig()
@kislyuk is there an easy way to dump the c14n payload? I have a document similar to @hathawsh that no longer validates because of this change without setting excise_empty_xmlns_declarations but I don't see an empty namespace declaration in his test to excise or not excise. There are some parent namespaces that get inherited by children but that seems different to me than an empty xmlns declaration.
@maxullman In the next minor release, I will be adding a Python standard library logger that will print all c14n payloads when its log level is set to DEBUG.