OpusCleaner

OpusCleaner copied to clipboard

Add notes to filter steps & data sets

1

Now I'm actually using the thing, I'm noticing an urge to document why I'm adding filters and why I set the parameters as I set them. But there's no proper...

jelmervdl

Any plans to share the filter configs?

5

It would be great to have full examples of the filters that were used to train a language pair. This would simplify the usage of the tool by non-expert users.

eu9ene

Support a hierarchy of filters

1

Hey, thank you for the great work as always! We're looking into integrating OpusFilter in Firefox Translations training pipeline! Our workflow is quite automated and most likely we'll keep our...

eu9ene

Workflow

1

Sorry bad title need to jot down some notes. Empty-train workflow, long version (maybe you can skip steps?) 1. Select datasets 1. Download each dataset 2. Generate samples 2. Select...

jelmervdl

Batch filter chain management

2

If you have many datasets, you'd want to apply the same filter steps to quite a couple of them I suspect. It would be helpful if we can provide some...

jelmervdl

How do I say this dataset is so garbage it shouldn't be used?

2

I downloaded a dataset, CCAligned-v1.en-mt. It has 37 sentences and maybe 2 are correct. How do I mark it as "do not use"?

kpu

langid vs fasttext_filter

1

I was surprised to learn that the `langid` filter is CLD2 and the `fasttext_filter` is fasttext langid. From a UI perspective, its better to have one. Aside, can we add...

kpu

What does the pie "show dataset statistics" do?

1

When I click on it, nothing happens.

kpu

1. Run fresh opuscleaner 2. Load http://localhost:8000 and get the empty folder (expected) 3. Click import dataset, select Maltese and English, click way too many buttons to download everything. 4....

kpu

Right now, opuscleaner is tricky to install because it will pull in all the dependencies for most filters. I'm tempted to remove most of the external filters (opusfilter, bicleaner, etc)...

jelmervdl