Luca Soldaini comments

Results 43 comments of


                                            Luca Soldaini

make_wikipedia.py: long running time

mmmh I have not tried to process DE Wikipedia in a while, but when I did it last year I was not having the same issue. I've heard good things...

Add PrefixSuffix multiseq mappers to prepend/append tokens

Hi Maksym! Thank you for this pull request. I fully support the reasoning behind adding these mappers, but I would prefer avoiding Mappers argument that are callables, as they can...

Support setting `strict=False` on specific fields

That would be a great feature!

Is there explicitly instruction-following data in the version of Dolma used to train v1?

(responded on ticket in Dolma repository)

SNOMED Vocab only?

Hi, Choosing which vocabularies to add should be done as part of the UMLS installation step (step 1 in the readme). You can definitely just install SNOMED or RXNORM that...

Is there explicitly instruction-following data in the version of Dolma used to train Olmo v1?

Hi John! Olmo V1 was trained on not explicitly trained on any instruction data. If any leak in occurs, I suspect is through the code subset we trained OLMo on....

Generation of own dataset with Dolma Tokenizer CLI

hey @WenJett ! You command looks correct, so it is strange that it is failing. is the `data.json.gz` something you could share?

Generation of own dataset with Dolma Tokenizer CLI

I tried with your file locally on my machine: ```shell dolma tokens --documents ./data.json.gz --destination ./ --tokenizer.name_or_path allenai/dolma2-tokenizer --tokenizer.eos_token_id 100257 --tokenizer.pad_token_id 100277 --dtype uint32``` ``` Checked the output as follows:...

Generation of own dataset with Dolma Tokenizer CLI

For issues with OLMo code, please open an issue on [its repo](https://github.com/allenai/olmo), referencing this issue. Thank you!