page__text.xsl is not honoring the reading order

[mikegerber]

page__text.xsl is not honoring the reading order in the PAGE-XML (pc:ReadingOrder), which gives completely false results. For this page, I get this text (shortened):

% docker run --rm -it -v "$PWD":/data ubma/ocr-fileformat ocr-transform page text OCR-D-GT-PAGE/OCR-D-GT-PAGE_00000024.xml | head
               
20
Die
[22.]
[22.]
ein gleies vorgegeben, und ſo gar ſehr viele mahle gegen ae menſlie Mglikeit mit Gewalt for-
ciret worden zu ſeyn, behaupten wi, mithin neb dem Bredekaw, weler (§. 28. 29.)  in aen ſeinen
Auagen wiederſproen, mit der Pœna fal um do gewier zu belegen i, da
ſecund. Farin. Tit. 9. qu. 66. p. m. 320.
die Klage ſo wohl als das Zeugnß vor falſ und erditet mßen gehalten werden.
§. 35) So viel die von der Inquitin
write /dev/stdout: broken pipe

For comparison, dinglehopper-extract gives the correct text:

% dinglehopper-extract OCR-D-GT-PAGE/OCR-D-GT-PAGE_00000024.xml| head
20
rath mit einer Pœna fiſcali angeſehen worden, und ſolche durch des Hrn. Graffen von Königsfeld Vor-
ſpruch, nur aus Gnaden nachgelaſſen erhalten.
Sondern man hat auch dieſen 4. Wochen lang alle Abend bey der Inquiſitin gantz allein gelaſſen.
Binnen welcher gantzer Zeit der Schreiber Bredekaw beſtändig bey Ihme geweſen, und ſich in
der am 13 ten Octobr. a.c. in Judicio gegen ſeinen geweſenen Hrn. introducirter Appellation deſſen Bey-
raths bedienet hat;
§. 33) Dabenebenſt iſt der Schreiber binnen dieſer gantzen Zeit auf freyem Fuß geblieben, und
hat nicht nur durch ſeinen Conſulenten, ſondern auch, weilen der Inquiſitin ſelbſten in Ihrem Gefängnüß
ſo viele Freyheit gelaſſen worden, daß ſie frembden Beſuch von Ihren Anverwandten ohngehindert em-

Image from the ZIP (converted to JPEG), for easier understanding:

[stweil]

The XSL which is used for that conversion is too simple to handle more complex PAGE XML. I don't know whether a better XSL is available from other projects.

Did you try whether JPageConverter which is already used for the PAGE to ALTO conversion does a better job? As far as I know it can also produce text.

[mikegerber]

I would consider this a serious bug, not an enhancement.

[stweil]

It's both. PAGE XML is complex, so I would never expect a perfect tool which supports all of its features.

[mikegerber]

It's not imperfection by not supporting some features, it's producing a wrong result if it's not honoring the reading order, for a lot of real world PAGE XML files.

[stweil]

The texts in the XML file also look strange when I look at them with less or vi ("Mglikeit"). Do you use some special encoding? It's not UTF-8!

[mikegerber]

The file in https://github.com/UB-Mannheim/ocr-fileformat/issues/138#issue-895785528 was created (by a SBB contractor) using Aletheia and uses their encoding scheme, which uses a lot of PUA characters, which in part is based on MUFI (See (https://www.primaresearch.org/www/assets/tools/Special%20Characters%20in%20Aletheia.pdf)). So it's UTF-8, but with private characters. But encoding is an entirely different beast :-) (dinglehopper-extract gives different characters due to normalization, but that's not the issue here.)

[bertsky]

Sry, did not see this earlier. But I had the exact same use case. It's not so difficult to properly handle PAGE reading order in XSLT 1.0. This was solved along with https://github.com/UB-Mannheim/ocr-fileformat/pull/151.

(You can even pass XSLT parameters for what hierarchy level you want to extract from (default is highest) or what separators to use for concatenation: https://github.com/UB-Mannheim/ocr-fileformat/blob/3e32ef632ff439710d123ba700364703d07b47a9/xslt/page__text.xsl#L14-L21

See

ocr-transform page text --help-args