pdfminer
CID characters when extracting text from Korean pdf
Hi,
I am testing a PDF file and when I try to run it using pdfminer.xi characters are broken and my pdf is encoded with /UniKS-UTF16-H This is the output coming (cid:53)(cid:51)(cid:53)(cid:54)(cid:15434)(cid:4738)(cid:11182)(cid:6530)(cid:35) (cid:11206)(cid:9838)(cid:11542)(cid:35) (cid:9967)(cid:4794)(cid:8882)(cid:4766)(cid:9946)
(cid:977)(cid:20) (cid:20) (cid:1923)
Here is my enviorment or pdfminer version
pip show pdfminer.six
Name: pdfminer.six Version: 20240706
Test PDF
code
from pdfminer.high_level import extract_text
file_path = "2023_._.9._.-7-12.pdf" # Replace with your PDF file path
text = extract_text(file_path, codec='UniKS-UTF16-H')
output_file = "output_pdfminer.md"
print("\n\n", text)
with open(output_file, "w", encoding="utf-16", errors="ignore") as md_file:
md_file.write(text)
print(f"Text extracted and saved as {output_file}")
Same problem as #1036 - again, try to copy and paste text out of the file and you will see that the mappings are just nonsense.
Not sure what's the problem, I copied text from the pdf and it indeed returns squares, but then I tried the same pdf with llamaparse and it returns text as in the pdf itself, could it be something else?
Oh, it could be that pdfminer has an old or broken version of UniKS-UTF16-H encoding - @mk-docenty can you try copying/pasting from Adobe Acrobat? (I just tried poppler and Chrome, which could have the same problems of incorrect encoding definitions)
Also @nnurmano this is the first I have heard of llamaparse. It appears to maybe be proprietary? Do you know what they are actually using to extract text from PDF?
Also @nnurmano this is the first I have heard of llamaparse. It appears to maybe be proprietary? Do you know what they are actually using to extract text from PDF?
No idea. But I shall try to find it out.
Some more digging in that PDF - the UniKS-UTF16-H encoding is only used by the font MalgunGothic, which is only used in the Form XObject on the first page. FontForge refuses to even open this font saying it is "corrupted beyond repair".
Most of the text in the tables (on page 3 for instance) is actually in IIBDIK+HCRDotum or IIBEIJ+GulimChe. These both use an "Identity" mapping - off the top of my head I don't recall exactly what this means, if anything, for text extraction. Here's GulimChe for instance:
67 0 obj
<<
/Ascent 858
/CIDSet 68 0 R
/CapHeight 0
/Descent -141
/Flags 4
/FontBBox [0 -150 1000 863]
/FontFile2 69 0 R
/FontName /IIBEIJ+GulimChe
/ItalicAngle 0
/StemV 0
/Type /FontDescriptor
/XHeight 0
>>
endobj
66 0 obj
<<
/BaseFont /IIBEIJ+GulimChe
/CIDSystemInfo <<
/Ordering (Identity)
/Registry (Adobe)
/Supplement 0
>>
/CIDToGIDMap /Identity
/DW 1000
/FontDescriptor 67 0 R
/Subtype /CIDFontType2
/Type /Font
/W [1659 [500] 1664 1685 500 1688 [500] 1692 1698 500 1700 [500] 1703 [500] 1705 1707 500 1709 1711 500 1714 1715 500 1718 [500] 1720 [500] 1722 1724 500 1731 [500] 1734 [500] 1736 [500] 1747 [500] 1750 [500] 1752 1753 500]
>>
endobj
65 0 obj
<<
/BaseFont /IIBEIJ+GulimChe
/DescendantFonts [66 0 R]
/Encoding /Identity-H
/Subtype /Type0
/Type /Font
>>
endobj
Looking at this in FontForge, it appears to have a collection of precomposed Hangul blocks and jamo at specific code points, which can be assumed to mean something, just not what pdfminer.six, PDFium and Poppler expect them to mean ;-) Do these code points mean anything to you?
I am writing a tool that reconstructs CID ToUnicode mapping. Verify.
2023_..9..-7-12.reconcstructed.pdf
I am writing a tool that reconstructs CID ToUnicode mapping. Verify.
not fixed here either
I am writing a tool that reconstructs CID ToUnicode mapping. Verify. 2023_..9..-7-12.reconcstructed.pdf
not fixed here either
Depends where. There are some cids remaining. But majority of the table is ok.
Exceprt:
선진회계행정 실현을 통해 도민의 삶의 질을 향상한다.