clarin-eric
validate-parlamint speedup
I have been exploring why the validation is so slow.
jing
jing allows to validation of multiple files with the same schema in parallel. These are the speeds for 64 thread CPU, in seconds:
| # of files | loading schema | validating | total time |
|---|---|---|---|
| 1 | .335 | .751 | 1.086 |
| 112 | .314 | 25.774 | 26.088 |
| 297 | .338 | 59.574 | 59.912 |
| 1149 | .335 | 24.8152 | 248.487 |
We can speed up jing 5 times, but the order of output will be different - not file by file.
We can speed up jing 5 times, but the order of output will be different - not file by file.
@TomazErjavec do we insist on this order?
I actually don't think jing is the bottleneck, rather, it is the XSLT validation that is slow. Also, validate-parlamint.pl takes file one by one, so it would be difficult to just do jing in parallel. In short, I don't think its worth trying to give jing multiple files.
I actually don't think jing is the bottleneck, rather, it is the XSLT validation that is slow. Also, validate-parlamint.pl takes file one by one, so it would be difficult to just do jing in parallel. In short, I don't think its worth trying to give jing multiple files.
I have tried it, and validate-parlamint is about 25% faster (tested on LV) with Jing passing multiple files to jing.
about 25% faster (tested on LV)
ok, but I still think it is not worth it given the other problems with this approach. This might save 10% processing time, if that.
with Jing passing multiple files to jing.
Huh?
Ok, I have staged my changes.
Another space for speeding up is the link-checker: Transform teiCorpus/teiHeader to a smaller temporary XML file, which contains just a list of elements with IDs - the parsing of this file can be faster, but the impact will be small too...
So no speedup, and moving to the future...
Another space for speeding up is the link-checker: Transform teiCorpus/teiHeader to a smaller temporary XML file, which contains just a list of elements with IDs - the parsing of this file can be faster, but the impact will be small too...
Yes, I think very small - the complete teiHeader (with everything XIncluded) fits into memory of any computer strong enough to process the corpus.
Parallel currently reports this warning on 60 threads:
INFO: Char validation for ParlaMint-IL_2004-01-01-16ptv487015.ana.xml
INFO: XML validation for ParlaMint-IL_2004-01-01-16ptv487015.ana.xml
INFO: XML validation for ParlaMint-IL_2004-01-01-16ptv487015.ana.xml
INFO: Content validaton for ParlaMint-IL_2004-01-01-16ptv487015.ana.xml
INFO: Link checking for ParlaMint-IL_2004-01-01-16ptv487015.ana.xml
parallel: Warning: No more file handles.
parallel: Warning: Try running 'parallel -j0 -N 100 --pipe parallel -j0'
parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
parallel: Warning: or increasing /proc/sys/fs/file-max
INFO: Validating component TEI.ana file /lnet/work/people/kopp/ParlaMint/Build/Distro/ParlaMint-IL.TEI.ana/2004/ParlaMint-IL_2004-01-04-16ptm533605.ana.xml
INFO: Char validation for ParlaMint-IL_2004-01-04-16ptm533605.ana.xml
INFO: XML validation for ParlaMint-IL_2004-01-04-16ptm533605.ana.xml
/lnet/work/people/kopp/ParlaMint/Build/Distro/ParlaMint-IL.TEI.ana/2004/ParlaMint-IL_2004-01-04-16ptm533605.ana.xml:122023:118: error: value of attribute "lemma" is invalid; must be a string matching the regular expression "(\S)|(\S[\S ]*\S)"
/lnet/work/people/kopp/ParlaMint/Build/Distro/ParlaMint-IL.TEI.ana/2004/ParlaMint-IL_2004-01-04-16ptm533605.ana.xml:1185883:106: error: value of attribute "lemma" is invalid; must be a string matching the regular expression "(\S)|(\S[\S ]*\S)"
/lnet/work/people/kopp/ParlaMint/Build/Distro/ParlaMint-IL.TEI.ana/2004/ParlaMint-IL_2004-01-04-16ptm533605.ana.xml:1364748:107: error: value of attribute "lemma" is invalid; must be a string matching the regular expression "(\S)|(\S[\S ]*\S)"
/lnet/work/people/kopp/ParlaMint/Build/Distro/ParlaMint-IL.TEI.ana/2004/ParlaMint-IL_2004-01-04-16ptm533605.ana.xml:1433903:104: error: value of attribute "lemma" is invalid; must be a string matching the regular expression "(\S)|(\S[\S ]*\S)"
INFO: XML validation for ParlaMint-IL_2004-01-04-16ptm533605.ana.xml
INFO: Content validaton for ParlaMint-IL_2004-01-04-16ptm533605.ana.xml
INFO: Link checking for ParlaMint-IL_2004-01-04-16ptm533605.ana.xml
INFO: Validating component TEI.ana file /lnet/work/people/kopp/ParlaMint/Build/Distro/ParlaMint-IL.TEI.ana/2004/ParlaMint-IL_2004-01-04-16ptv547034.ana.xml
it seems that it does not influence the validation process