Feature Request: automatic conversion between Elixir Binary and `Vec<u8>` when using serde

[yizhepku]

Due to limitations of serde, Vec<u8> will serialize into "seq of u8", which is rendered into Elixir list instead of binary. There are libraries that force serde into treating Vec<u8> as bytes (e.g. serde_bytes, serde_with), but these libraries require users to annotate existing Rust code, which is not always possible or convenient.

Given that the goal of rustler is to make interop between Elixir and Rust as smooth as possible, I believe encoding Vec<u8> into binaries is a better default.

I propose the following breaking changes to rustler's serde conversion rules:

1. Allow Elixir binary to be deserialized into "seq of u8". This means SerdeTerm<Vec<u8>> can be decoded from binary.
2. Prefer to serialize "seq of u8" into binary when possible. This means SerdeTerm<Vec<u8>> will be encoded into binary, but SerdeTerm<Vec<i8>> will be encoded into list, as usual.

I'm currently working on an implementation. Any thoughts?

[yizhepku]

Now that I look at rustler's conversion table again, doesn't ruslter already claim that it encodes Vec<u8> into binary? If so, this might just be a bug fix rather than a new feature.

[filmor]

I'll have a look. The current serde suppprt is, apart from major refactorings and adjustments to make it work with modern rustler, basically still equivalent to serde_rustler. I didn't work on verifying or adjusting conversion rules, yet.

I would really like to be able to configure rustler in general and the serde support specifically at init time (which would also encompass a solution to #667), but I haven't gotten around prototyping this.