Zengzhi Wang

icon indicating a website link https://sinclaircoder.github.io/

icon location China icon location PhD student. #NLProc A data engineering boy: MathPile (2023), 🫐 ProX (2024)

Results 14 comments of Zengzhi Wang

Problem with invalid Mathmode "AssertionError: Command \item invalid in math mode."

how to solve it?

Link in Readme produces 404

It also produces 404!

Link in Readme produces 404

So how to download this dataset now?

Link in Readme produces 404

ok

data in modelscope for more convenient to download?

Thanks for your feedback. We're thinking about it. You can also download it through the [HF mirror](https://hf-mirror.com/) if issued by the network issue.

data in modelscope for more convenient to download?

hi, MathPile is also available at Wisemodel now! Feel free to download it! https://wisemodel.cn/datasets/GAIR/MathPile https://wisemodel.cn/datasets/GAIR/MathPile_Commercial

Dataset code

Thank you very much for your valuable feedback. We are currently dedicating our efforts to refining our datasets and enhancing the quality of our data, alongside progressing with model training....

Dataset code

Scripts for preprocessing and global data processing have been updated. Please feel free to take a look at it.

Memory overhead in multiprocessing

Hi, is your problem solved now? I also encountered some similar issues (unexpected OOM resulting in failed jobs). I also think the source of the unexpected OOM issues may be...

Memory overhead in multiprocessing

My OOM case happened in the text extractor. But I do not know how to fix it. Sad.