Luca Soldaini comments

Results 43 comments of


                                            Luca Soldaini

Is there a way to intergratge Dolma toolkit to Spark?

hi @DangoWang. I haven't investigated spark integration, but you should be able to wrap individual taggers in spark.

S3 mixer doesn't start

hello @marcopasqua! Could you post your mixer configuration? you might have a typo there. I suggest running the mix command with `--dryrun` to print the configuration as it is parsed...

make_wikipedia.py fails on linux

Hi @peterbjorgensen! thank you for this bug report. I've made a PR (#64) with these fixes in. I can't seem to reproduce the error gzip... could you tell me a...

make_wikipedia.py fails on linux

I remain unable to reproduce this issue on my side, would need more info.

Data out of bounds when using ‘dolma tokens --dtype uint32’

Thank you for the report @Jackwaterveg. Could you re-run the command above with `--dryrun` to show the full configuration? thanks.

Some race condition in url taggers

Hi @peterbjorgensen! which taggers are you trying to run? More information for me to reproduce would be great.

not_alphanum_paragraph_v1 tagger takes forever to run on certain inputs.

Yikes. Probably the easiest way to tackle this is to create two version of the taggers; one using regex, the other using re.

Change bloom_filter implementation of hash

Tagging @dirkgr since bloom filter is his baby.

Change bloom_filter implementation of hash

I am okay with merging in #39, but yeah the goal is to eventually merge back into the BFF repo and release that as a crate. @dirkgr given your schedule,...

Change bloom_filter implementation of hash

If you don't mind fixing the racing condition, I'll be happy to merge a PR. I don't think we have strong opinions about which solution to use, so I'm happy...