dolma icon indicating copy to clipboard operation
dolma copied to clipboard

Published 20 hours ago • verified publisher icon allenai

Text modification config

Open rodneykinney opened this issue 8 months ago • 1 comments

Add mixer configuration to trim trailing/leading whitespace from document text, and enforce a minimum document text length. Place these into a new text_modification config object, and move the span_replacements config into it.

@soldni any objections to this backward-incompatible change to config structure?

rodneykinney avatar Oct 19 '23 16:10 rodneykinney