dumbo icon indicating copy to clipboard operation
dumbo copied to clipboard

Reading text as typedbytes affects lines with encoding other than utf8

Open a4tunado opened this issue 10 years ago • 0 comments

Streaming backend assumes that input format is typedbytes even if -inputformat argument is 'text': https://github.com/klbostee/dumbo/blob/release-0.21.36/dumbo/backends/streaming.py#L81

This leads to apply typedbytes.PairedInput to all input lines: https://github.com/klbostee/dumbo/blob/release-0.21.36/dumbo/core.py#L380

Appling util.loadtext instead of typedbytes.PairedInput resolves this issue.

a4tunado avatar Mar 01 '14 12:03 a4tunado