pycorrector icon indicating copy to clipboard operation
pycorrector copied to clipboard

Published 20 hours ago • verified publisher icon shibing624

大规模语料训练kenlm内存泄漏

Open ZTurboX opened this issue 3 years ago • 5 comments

用大规模语料训练kenlm，使用模型时会内存泄漏

ZTurboX avatar Apr 18 '22 07:04 ZTurboX

是内存爆了吗？还是kenlm代码本身的原因

shibing624 avatar Apr 18 '22 12:04 shibing624

maybe this help you https://github.com/kpu/kenlm/issues/185

shibing624 avatar Apr 18 '22 12:04 shibing624

训练时可以使用-S 参数来控制内存使用，例如：-S 10%

sunM123 avatar Apr 22 '22 07:04 sunM123

是内存爆了吗？还是kenlm代码本身的原因

预测时，大量请求，内存会爆满，很久不会释放

ZTurboX avatar Apr 22 '22 09:04 ZTurboX

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.(由于长期不活动，机器人自动关闭此问题，如果需要欢迎提问)

stale[bot] avatar Aug 13 '22 08:08 stale[bot]