pypdf PdfMerger breaks PDF/A compliance

PdfMerger breaks PDF/A compliance

Open MartinThoma opened this issue 2 years ago • 5 comments

Use PdfMerger with a single PDF/A compliant document I would expect almost exactly the same output file as the input file. But it's way different - and PDF/A compliance is broken.

Code + PDF

Using this as an example document: https://www.pdfa.org/wp-content/uploads/2011/08/PDFA-in-a-Nutshell_1b.pdf

And https://demo.verapdf.org/ to verify if the document is compliant.

from PyPDF2 import PdfReader, PdfMerger

reader = PdfReader("PDFA-in-a-Nutshell_1b.pdf")
metadata = reader.metadata

merger = PdfMerger()
merger.append(reader)
merger.add_metadata(metadata)

with open("merged.pdf", "wb") as fp:
    merger.write(fp)

Issues

PDFA-in-a-Nutshell_1b.pdf has 6.4 MB and is PDF/A compliant
merged.pdf has 5.0 MB and is NOT PDF/compliant

verapdf.org mentions that 100 issues were detected. It lists the following 3:

The document catalog dictionary of a conforming file shall contain the Metadata key. (docs) x1
DeviceCMYK may be used only if the file has a PDF/A-1 OutputIntent that uses a CMYK colour space (docs) x97
The file trailer dictionary shall contain the ID keyword. The file trailer referred to is either the last trailer dictionary in a PDF file, as described in PDF Reference 3.4.4 and 3.4.5, or the first page trailer in a linearized PDF file, as described in PDF Reference F.2 (docs) x1
If a document information dictionary does appear at a document, then all of its entries that have analogous properties in predefined XMP schemas, shall also be embedded in the file in XMP form with equivalent values. (docs) x1

Jun 19 '22 18:06 MartinThoma

The metadata section that is missing looks like this in the original file:

1 0 obj
<< /Metadata 3 0 R /Outlines 4 0 R /OutputIntents [ << /DestOutputProfile 5 0 R /Info (ISO Coated v2 \(ECI\)) /OutputConditionIdentifier (ISO Coated v2 \(ECI\)) /RegistryName (http://www.color.org) /S /GTS_PDFX /Type /OutputIntent >> << /DestOutputProfile 5 0 R /Info (ISO Coated v2 \(ECI\)) /OutputConditionIdentifier (ISO Coated v2 \(ECI\)) /RegistryName (http://www.color.org) /S /GTS_PDFA1 /Type /OutputIntent >> ] /PageLabels 6 0 R /Pages 7 0 R /Type /Catalog /ViewerPreferences << /Direction /L2R >> >>
endobj
2 0 obj
<< /Author (PDF/A Competence Center) /CreationDate (D:20110818145925+02'00') /Creator (Adobe InDesign CS5 \(7.0.4\)) /GTS_PDFXConformance (PDF/X-1a:2003) /GTS_PDFXVersion (PDF/X-1a:2003) /ModDate (D:20110818150035+02'00') /Producer (Adobe PDF Library 9.9) /Title (PDF/A in a Nutshell) /Trapped /False >>
endobj
3 0 obj
<< /Subtype /XML /Type /Metadata /Length 12889 >>
stream
<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 4.2.1-c043 52.372728, 2009/01/18-15:56:37        ">
   <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
      <rdf:Description rdf:about=""
            xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/"
            xmlns:stEvt="http://ns.adobe.com/xap/1.0/sType/ResourceEvent#">
         <xmpMM:InstanceID>uuid:2b5a9f85-f518-7b4a-a756-79898ed7b891</xmpMM:InstanceID>
         <xmpMM:OriginalDocumentID>adobe:docid:indd:fbe35371-5d32-11dc-b86a-8404c5e05271</xmpMM:OriginalDocumentID>
         <xmpMM:DocumentID>adobe:docid:indd:fbe35371-5d32-11dc-b86a-8404c5e05271</xmpMM:DocumentID>
         <xmpMM:RenditionClass>proof:pdf</xmpMM:RenditionClass>
         <xmpMM:VersionID>1</xmpMM:VersionID>
         <xmpMM:History>
            <rdf:Seq>
               <rdf:li rdf:parseType="Resource">
                  <stEvt:action>converted</stEvt:action>
                  <stEvt:instanceID>uuid:d6a6d0f6-aead-f743-bda5-13d4cf5c3c11</stEvt:instanceID>
                  <stEvt:parameters>converted to PDF/A-1b</stEvt:parameters>
                  <stEvt:softwareAgent>pdfaPilot</stEvt:softwareAgent>
                  <stEvt:when>2011-08-18T15:00:32+02:00</stEvt:when>
               </rdf:li>
            </rdf:Seq>
         </xmpMM:History>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:xmp="http://ns.adobe.com/xap/1.0/">
         <xmp:CreatorTool>Adobe InDesign CS5 (7.0.4)</xmp:CreatorTool>
         <xmp:CreateDate>2011-08-18T14:59:25+02:00</xmp:CreateDate>
         <xmp:ModifyDate>2011-08-18T15:00:35+02:00</xmp:ModifyDate>
         <xmp:MetadataDate>2011-08-18T15:00:35+02:00</xmp:MetadataDate>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:dc="http://purl.org/dc/elements/1.1/">
         <dc:format>application/pdf</dc:format>
         <dc:title>
            <rdf:Alt>
               <rdf:li xml:lang="x-default">PDF/A in a Nutshell</rdf:li>
            </rdf:Alt>
         </dc:title>
         <dc:creator>
            <rdf:Seq>
               <rdf:li>PDF/A Competence Center</rdf:li>
            </rdf:Seq>
         </dc:creator>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:pdf="http://ns.adobe.com/pdf/1.3/">
         <pdf:Producer>Adobe PDF Library 9.9</pdf:Producer>
         <pdf:Trapped>False</pdf:Trapped>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:pdfxid="http://www.npes.org/pdfx/ns/id/">
         <pdfxid:GTS_PDFXVersion>PDF/X-1a:2003</pdfxid:GTS_PDFXVersion>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:pdfx="http://ns.adobe.com/pdfx/1.3/">
         <pdfx:GTS_PDFXVersion>PDF/X-1a:2003</pdfx:GTS_PDFXVersion>
         <pdfx:GTS_PDFXConformance>PDF/X-1a:2003</pdfx:GTS_PDFXConformance>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:pdfaid="http://www.aiim.org/pdfa/ns/id/">
         <pdfaid:part>1</pdfaid:part>
         <pdfaid:conformance>B</pdfaid:conformance>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:pdfaExtension="http://www.aiim.org/pdfa/ns/extension/"
            xmlns:pdfaSchema="http://www.aiim.org/pdfa/ns/schema#"
            xmlns:pdfaProperty="http://www.aiim.org/pdfa/ns/property#">
         <pdfaExtension:schemas>
            <rdf:Bag>
               <rdf:li rdf:parseType="Resource">
                  <pdfaSchema:namespaceURI>http://ns.adobe.com/pdf/1.3/</pdfaSchema:namespaceURI>
                  <pdfaSchema:prefix>pdf</pdfaSchema:prefix>
                  <pdfaSchema:schema>Adobe PDF</pdfaSchema:schema>
                  <pdfaSchema:property>
                     <rdf:Seq>
                        <rdf:li rdf:parseType="Resource">
                           <pdfaProperty:category>internal</pdfaProperty:category>
                           <pdfaProperty:description>A name object indicating whether the document has been modified to include trapping information</pdfaProperty:description>
                           <pdfaProperty:name>Trapped</pdfaProperty:name>
                           <pdfaProperty:valueType>Text</pdfaProperty:valueType>
                        </rdf:li>
                     </rdf:Seq>
                  </pdfaSchema:property>
               </rdf:li>
               <rdf:li rdf:parseType="Resource">
                  <pdfaSchema:namespaceURI>http://ns.adobe.com/pdfx/1.3/</pdfaSchema:namespaceURI>
                  <pdfaSchema:prefix>pdfx</pdfaSchema:prefix>
                  <pdfaSchema:schema>PDF/X ID Schema</pdfaSchema:schema>
                  <pdfaSchema:property>
                     <rdf:Seq>
                        <rdf:li rdf:parseType="Resource">
                           <pdfaProperty:category>internal</pdfaProperty:category>
                           <pdfaProperty:description>ID of PDF/X standard</pdfaProperty:description>
                           <pdfaProperty:name>GTS_PDFXVersion</pdfaProperty:name>
                           <pdfaProperty:valueType>Text</pdfaProperty:valueType>
                        </rdf:li>
                        <rdf:li rdf:parseType="Resource">
                           <pdfaProperty:category>internal</pdfaProperty:category>
                           <pdfaProperty:description>Conformance level of PDF/X standard</pdfaProperty:description>
                           <pdfaProperty:name>GTS_PDFXConformance</pdfaProperty:name>
                           <pdfaProperty:valueType>Text</pdfaProperty:valueType>
                        </rdf:li>
                        <rdf:li rdf:parseType="Resource">
                           <pdfaProperty:category>internal</pdfaProperty:category>
                           <pdfaProperty:description>Company creating the PDF</pdfaProperty:description>
                           <pdfaProperty:name>Company</pdfaProperty:name>
                           <pdfaProperty:valueType>Text</pdfaProperty:valueType>
                        </rdf:li>
                        <rdf:li rdf:parseType="Resource">
                           <pdfaProperty:category>internal</pdfaProperty:category>
                           <pdfaProperty:description>Date when document was last modified</pdfaProperty:description>
                           <pdfaProperty:name>SourceModified</pdfaProperty:name>
                           <pdfaProperty:valueType>Text</pdfaProperty:valueType>
                        </rdf:li>
                     </rdf:Seq>
                  </pdfaSchema:property>
               </rdf:li>
               <rdf:li rdf:parseType="Resource">
                  <pdfaSchema:namespaceURI>http://ns.adobe.com/xap/1.0/mm/</pdfaSchema:namespaceURI>
                  <pdfaSchema:prefix>xmpMM</pdfaSchema:prefix>
                  <pdfaSchema:schema>XMP Media Management</pdfaSchema:schema>
                  <pdfaSchema:property>
                     <rdf:Seq>
                        <rdf:li rdf:parseType="Resource">
                           <pdfaProperty:category>internal</pdfaProperty:category>
                           <pdfaProperty:description>UUID based identifier for specific incarnation of a document</pdfaProperty:description>
                           <pdfaProperty:name>InstanceID</pdfaProperty:name>
                           <pdfaProperty:valueType>URI</pdfaProperty:valueType>
                        </rdf:li>
                        <rdf:li rdf:parseType="Resource">
                           <pdfaProperty:category>internal</pdfaProperty:category>
                           <pdfaProperty:description>The common identifier for all versions and renditions of a document.</pdfaProperty:description>
                           <pdfaProperty:name>OriginalDocumentID</pdfaProperty:name>
                           <pdfaProperty:valueType>URI</pdfaProperty:valueType>
                        </rdf:li>
                     </rdf:Seq>
                  </pdfaSchema:property>
               </rdf:li>
               <rdf:li rdf:parseType="Resource">
                  <pdfaSchema:namespaceURI>http://www.aiim.org/pdfa/ns/id/</pdfaSchema:namespaceURI>
                  <pdfaSchema:prefix>pdfaid</pdfaSchema:prefix>
                  <pdfaSchema:schema>PDF/A ID Schema</pdfaSchema:schema>
                  <pdfaSchema:property>
                     <rdf:Seq>
                        <rdf:li rdf:parseType="Resource">
                           <pdfaProperty:category>internal</pdfaProperty:category>
                           <pdfaProperty:description>Part of PDF/A standard</pdfaProperty:description>
                           <pdfaProperty:name>part</pdfaProperty:name>
                           <pdfaProperty:valueType>Integer</pdfaProperty:valueType>
                        </rdf:li>
                        <rdf:li rdf:parseType="Resource">
                           <pdfaProperty:category>internal</pdfaProperty:category>
                           <pdfaProperty:description>Amendment of PDF/A standard</pdfaProperty:description>
                           <pdfaProperty:name>amd</pdfaProperty:name>
                           <pdfaProperty:valueType>Text</pdfaProperty:valueType>
                        </rdf:li>
                        <rdf:li rdf:parseType="Resource">
                           <pdfaProperty:category>internal</pdfaProperty:category>
                           <pdfaProperty:description>Conformance level of PDF/A standard</pdfaProperty:description>
                           <pdfaProperty:name>conformance</pdfaProperty:name>
                           <pdfaProperty:valueType>Text</pdfaProperty:valueType>
                        </rdf:li>
                     </rdf:Seq>
                  </pdfaSchema:property>
               </rdf:li>
               <rdf:li rdf:parseType="Resource">
                  <pdfaSchema:namespaceURI>http://www.npes.org/pdfx/ns/id/</pdfaSchema:namespaceURI>
                  <pdfaSchema:prefix>pdfxid</pdfaSchema:prefix>
                  <pdfaSchema:schema>PDF/X ID Schema</pdfaSchema:schema>
                  <pdfaSchema:property>
                     <rdf:Seq>
                        <rdf:li rdf:parseType="Resource">
                           <pdfaProperty:category>internal</pdfaProperty:category>
                           <pdfaProperty:description>ID of PDF/X standard</pdfaProperty:description>
                           <pdfaProperty:name>GTS_PDFXVersion</pdfaProperty:name>
                           <pdfaProperty:valueType>Text</pdfaProperty:valueType>
                        </rdf:li>
                     </rdf:Seq>
                  </pdfaSchema:property>
               </rdf:li>
            </rdf:Bag>
         </pdfaExtension:schemas>
      </rdf:Description>
   </rdf:RDF>
</x:xmpmeta>

Jun 19 '22 18:06 MartinThoma

The file trailer of the original looks like this:

trailer << /Info 2 0 R /Root 1 0 R /Size 810 /ID [<cc24aff220034a578f97b292897ecfb3><482975899621bd41f614731fdea45046>] >>

whereas the merged one looks like this:

trailer << /Info 2 0 R /Root 1 0 R /Size 783 /ID [<1916dc78292f2472ad342a4f13641c7f><1916dc78292f2472ad342a4f13641c7f>] >>

So we do have the ID keyword, but we violate

(isLinearized == true && firstPageID != null) || ((isLinearized != true) && lastID != null)

Jun 19 '22 18:06 MartinThoma

https://avepdf.com/pdfa-validation might also help us

Jun 26 '22 07:06 MartinThoma

This probably should be updated to reflect the deprecation of PdfMerger in favor PdfWriter.

Dec 24 '23 20:12 stefan6419846

According to pdfinfo PDFA-in-a-Nutshell_1b.pdf states its conformance as Level A, Accessible, as well as the generated PDF file:

>>> from pypdf import PdfReader, PdfWriter
>>> reader = PdfReader('PDFA-in-a-Nutshell_1b.pdf')
>>> metadata = reader.metadata
>>> writer = PdfWriter(clone_from=reader)
>>> writer.add_metadata(metadata)
>>> writer.write('merged.pdf')
(True, <_io.FileIO [closed]>)
>>>

Running this through VeraPDF with the PDF/A-1A profile, I get some different issues:

The document catalog dictionary shall include a MarkInfo dictionary with a Marked entry in it, whose value shall be true. (docs) ×1
The font dictionary shall include a ToUnicode entry whose value is a CMap stream object that maps character codes to Unicode values, as described in PDF Reference 5.9, unless the font meets any of the following three conditions: () fonts that use the predefined encodings MacRomanEncoding, MacExpertEncoding or WinAnsiEncoding, or that use the predefined Identity-H or Identity-V CMaps; () Type 1 fonts whose character names are taken from the Adobe standard Latin character set or the set of named characters in the Symbol font, as defined in PDF Reference Appendix D; (*) Type 0 fonts whose descendant CIDFont uses the Adobe-GB1, Adobe-CNS1, Adobe-Japan1 or Adobe-Korea1 character collections. (docs) ×65
The logical structure of the conforming file shall be described by a structure hierarchy rooted in the StructTreeRoot entry of the document catalog dictionary, as described in PDF Reference 9.6. (docs) ×1
A Level A conforming file shall specify the value of pdfaid:conformance as A. (docs) ×1
The file trailer dictionary shall contain the ID keyword. The file trailer referred to is either the last trailer dictionary in a PDF file, as described in PDF Reference 3.4.4 and 3.4.5, or the first page trailer in a linearized PDF file, as described in PDF Reference F.2. (docs) ×1

Using the automatically detected profile (PDF/A-1B) only item 5 is being reported.

Dec 29 '23 13:12 stefan6419846

pypdf pypdf copied to clipboard

PdfMerger breaks PDF/A compliance

Code + PDF

Issues

pypdf
pypdf copied to clipboard