fastBPE icon indicating copy to clipboard operation
fastBPE copied to clipboard

Published 20 hours ago • verified publisher icon glample

learnbpe and the process is killed

Open tuanshanyou opened this issue 5 years ago • 1 comments

the corpus is 4G,and the memory is 16G i guess that it's KILLED because the memory is full. how to deal with it if it do not reduce the ncodes?

tuanshanyou avatar Jul 19 '19 07:07 tuanshanyou

@tuanshanyou , you can try this new library: https://github.com/VKCOM/YouTokenToMe. This library is more efficient than fastBPE.

Oktai15 avatar Jul 19 '19 14:07 Oktai15