ASCII-8BIT to UTF-8 [Encoding::UndefinedConversionError]

[benamormed98]

---

/usr/lib/ruby/3.0.0/json/common.rb:406:in generate' /usr/lib/ruby/3.0.0/json/common.rb:406:in pretty_generate' /var/lib/gems/3.0.0/gems/oxidized-0.29.1/lib/oxidized/output/http.rb:50:in generate_json' /var/lib/gems/3.0.0/gems/oxidized-0.29.1/lib/oxidized/output/http.rb:31:in store' /var/lib/gems/3.0.0/gems/oxidized-0.29.1/lib/oxidized/worker.rb:65:in process_success' /var/lib/gems/3.0.0/gems/oxidized-0.29.1/lib/oxidized/worker.rb:47:in process' /var/lib/gems/3.0.0/gems/oxidized-0.29.1/lib/oxidized/worker.rb:16:in block in work' /var/lib/gems/3.0.0/gems/oxidized-0.29.1/lib/oxidized/worker.rb:16:in each' /var/lib/gems/3.0.0/gems/oxidized-0.29.1/lib/oxidized/worker.rb:16:in work' /var/lib/gems/3.0.0/gems/oxidized-0.29.1/lib/oxidized/core.rb:36:in run' /var/lib/gems/3.0.0/gems/oxidized-0.29.1/lib/oxidized/core.rb:29:in initialize' /var/lib/gems/3.0.0/gems/oxidized-0.29.1/lib/oxidized/core.rb:4:in new' /var/lib/gems/3.0.0/gems/oxidized-0.29.1/lib/oxidized/core.rb:4:in new' /var/lib/gems/3.0.0/gems/oxidized-0.29.1/lib/oxidized/cli.rb:13:in run' /var/lib/gems/3.0.0/gems/oxidized-0.29.1/bin/oxidized:8:in <top (required)>' /usr/local/bin/oxidized:25:in load' /usr/local/bin/oxidized:25:in `

'

[benamormed98]

Hello,

can someone knows to fix this issue !

---

resolve_dns: true interval: 0 use_syslog: false threads: 30 timeout: 1200 retries: 2 prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/ rest: 0.0.0.0:5000 next_adds_job: false vars: remove_secret: false groups: {} models: {} pid: "/home/oxidized/.config/oxidized/pid" crash: directory: "/home/oxidized/.config/oxidized/crashes" input: default: ssh debug: false ssh: secure: false output: default: http http: url: "http://xxxx" source: default: sql debug: false sql: adapter: sqlite database: "/var/lib/oxidized/nodes.db" table: nodes map: group: group ip: ip model: model name: name password: password username: username

[benamormed98]

when i use output : default file there are no problem , but when using output default : http i got this error

[benamormed98]

I found how to fix it ; we have to modify the function generate_json

def generate_json(node, outputs, opt) config_data = outputs.to_cfg.force_encoding('UTF-8') JSON.pretty_generate( 'msg' => opt[:msg], 'user' => opt[:user], 'email' => opt[:email], 'group' => opt[:group], 'node' => node, 'config' => config_data # actually we need to also iterate outputs, for other types like in gitlab. But most people don't use 'type' functionality. ) end

[ytti]

Mohamed emailed me directly and I think the solution for this is likely highly unsatisfactory.

The example that was shared the non-UTF8 byte was 0xC3. The solution @benamormed98 suggest, will break that byte, by forcing UTF-8 encoding, so @benamormed98 no longer can restore 0xC3 byte from the config, potentially breaking the configuration restore.

We can review two cases why 0xC3 is in the config:

Case1) 0xC3 is legitimate byte which must be stored:

• We cannot change the HTTP output, as we will break all existing implementation
• We'd need some option to base64 code the config, before creating JSON, and the remote end would have to support those base64 encoding - needs specific changes in remote-end too!

Case2) 0xC3 is illegitimate byte, mistakenly in config:

• The config must be fixed
• Likely reason, someone logged to the device, typed some interface description or something using local non-UTF8 8bit encoding, where '0xC3' is some non ascii-7bit character from local alphabet

For git output we store 'bytes', so any crap we can store. But we can't do this with JSON, we can't put non-UTF8 in JSON. And if we use base64 in JSON, then far-end has to specifically support turning the base64 back into 8bit base string.