Memory offset shouldn't be influenced by the reading byte size?

[myobie]

trafficstars

I've got a situation where I need to start reading uint32's in memory starting at a byte offset like 555. However, Memory.get/4 expects the entire memory stream to be broken down into elements of the chosen element_size instead of "all bytes after the offset" which makes it impossible to read uint32's from memory unless the offset is exactly divisible by 4.

So Memory.get(memory, :uint32, 555, 0) where 555 is the byte offset is not possible.

• Did I explain that well?
• Is it alright to change the existing functions to accept a "byte offset" instead or should there be new functions?
• Do you have an idea how to achieve this in the rust code other than to always start with the <u8> view and then recalculate the view from there?

[tessi]

Thanks for opening this issue! I think you explained well (at least I have the feeling of understanding :D)

Is it alright to change the existing functions to accept a "byte offset" instead or should there be new functions?

We are still in the development phase and haven't stabilized yet (whatever that means). Changing the behavior of that function is OK as long as we properly update the package version and CHANGELOG.

Do you have an idea how to achieve this in the rust code other than to always start with the view and then recalculate the view from there?

Good question. We may have to look into wasmer internals - which, at this point in time, may not be the best idea. They are rewriting wasmer from scratch (see #45 ). I got access to a beta tester preview of their rewrite and currently adapt wasmex to work with it (it does not compile yet, sadly).

We could totally fix this issue now (feel free to attempt). But if you can wait for the rewrite, it may save some effort to start with the new version once it's available.

[myobie]

I have some code to work around it for now by doing the integer conversion in elixir:

  @spec get_uint32(Wasmex.Memory.t(), non_neg_integer, non_neg_integer) :: non_neg_integer
  defp get_uint32(memory, offset, index) do
    <<value::little-32>> =
      for i <- 0..3, into: <<>>, do: <<Wasmex.Memory.get(memory, :uint8, offset, index + i)>>

    value
  end

so I am fine with waiting a bit to change the rust. I am wondering if it makes sense to just do this work in elixir and have rust always see the memory as bytes? @tessi what do you think about that?

[tessi]

Hmm, I think (if possible to do) offset should always be bytes. If we do all the bytes-to-actual-values-conversion in elixir, I fear of performance impact for large chunks of memory.

If we're done with adapting to the wasmer rewrite, I will try to make offset count bytes instead of arbitrary units. If that does not work, conversion in elixir is probably the next best solution.

[myobie]

@tessi I think your idea of the offset always being in bytes makes sense and it would work for my use case. I can keep using my elixir code until you are ready to work on this :bow:

[tessi]

@myobie I gave the whole memory API some thinking over the last couple of days and the more I look at it, the more I feel we shouldn't support all the different types (u8, u16, u32, ...) in the first place. At the wasmex layer, I think, we should closely follow WASMs linear memory model of a memory just being a couple of bytes.

In my experiment to rewrite wasmex to the wasmtime engine, I already made that change. And the memory API is soo much simpler and more consistent.

To properly interface with WASM programs, my feeling is the Elixir app that uses wasmex needs to decide for the ABI with the WASM program. This is not something wasmex can or should do 🤔 We can only make it simple by having clear APIs.

Maybe, we can solve that by having some elixir-side helper methods, or (very long term) utilize something like https://github.com/bytecodealliance/wit-bindgen to generate Elixir code that interfaces with WASM. Maybe something close to what wasm-bindgen is for the JS ecosystem.

What do you think?

[myobie]

@tessi I think just a list of bytes is perfect.

For my use case, I can't use bindgen since I use the wasm binary on a lot of different platforms and not every platform has a bindgen, so my exported wasm functions are expected to be used directly and communicate with either ints or floats.

[tessi]

Since we're preparing the aforementioned wasmtime switch (having a different memory API), I believe this ticket can be closed.

Nathan, please feel free to reopen if you feel this is undeserved :)