你好，请问质检层相关的内容很感兴趣，你们有什么计划吗？

[xuquankun]

[drunkpig]

Regarding the quality assurance of large-scale mixed PDFs, we proceed as follows:

1. Manually select several categories (such as papers, textbooks, exam papers, research reports, etc.) from PDF pages we have processed, not limited to these, and manually pick out the pages we have processed well. The more diversity, the better.
2. Convert these PDF pages into images.
3. These images are extracted into a feature library (using ResNet50).
4. When a PDF page of unknown quality is extracted, use the screenshot of this page to query the most similar PDF pages from the feature library and record the distance.
5. As a batch of Pdfs processing concludes, we filter out those pages whose distances are in the bottom N% and analyze them one by one to improve our model or drop these data.

@xuquankun

[xuquankun]

我理解了你们这块工作的流程，请问质检工具你们会开发开源吗？

[drunkpig]

@xuquankun This functionality relies on continuously data accumulating , and the tools are not fully developed. Some steps are completed by different team members using various scripts, so it is not suitable for open sourcing

[xuquankun]

明白了，谢谢