morphpiece comments

Repositories
Issues
Comments

Results 4 comments of


                                            morphpiece

"Solution" to memory hogging in train_new_from_iterator with a hack

Can you give a short script to reproduce the problem?

"Solution" to memory hogging in train_new_from_iterator with a hack

I just ran the script and didn't observe any OOM. I used a 30 vCPU/240 GB AMD-Epyc server. The memory consumption increased linearly during count_pairs stage and the max was...

is max_seq_len configurable or hardcoded parameter?

Two points come to mind. It's fine to hardcode at this stage (like vocabulary), but then it should **not** be available as a configurable parameter. That can throw someone off...

Using the Authenticator with credentials stored in a remote database

@felipecordero Can you maybe make a pull request of your implementation? Or put a link here, so that others can use it as well. Thanks