`Memory` with more than 32767 pages cannot be accessed

[kpreisser]

Hi, consider the following C# program (.NET 6.0, using Wasmtime 1.0.0):

using var engine = new Engine();

using var module1 = Module.FromText(
    engine,
    "hello",
    @"
(module 
    (memory $mem1 65536)
    (export ""memory"" (memory $mem1))
)");

using (var linker = new Linker(engine)) {
    using var store = new Store(engine);
    var instance = linker.Instantiate(store, module1);

    var memory = instance.GetMemory("memory")!;

    try {
        memory.WriteInt16(10, 123);
        Console.WriteLine("Success");
    }
    catch (Exception ex) {
        Console.WriteLine(ex.ToString());
    }

    try {
        var memorySpan = memory.GetSpan();
        Console.WriteLine("Success");
    }
    catch (Exception ex) {
        Console.WriteLine(ex.ToString());
    }
}

Actual behavior: When running the program, an OverflowException occurs when the memory is accessed. E.g. with Memory.WriteInt16():

System.OverflowException: Arithmetic operation resulted in an overflow.
   at System.UIntPtr.ToUInt32()
   at Wasmtime.Memory.GetSpan() in C:\Users\VPC\Desktop\WebAssembly-Test\wasmtime-dotnet\src\Memory.cs:line 92
   at Wasmtime.Memory.WriteInt16(Int32 address, Int16 value) in C:\Users\VPC\Desktop\WebAssembly-Test\wasmtime-dotnet\src\Memory.cs:line 228
   at Program.<<Main>$>g__TestBigMemoryCustomWasm|0_3() in C:\Users\VPC\Desktop\WebAssembly-Test\TestWasmtime\Program.cs:line 326

Or, when changing the memory definition to (memory $mem1 32768), a similar OverflowException occurs:

System.OverflowException: Value was either too large or too small for an Int32.
   at System.Convert.ThrowInt32OverflowException()
   at System.Convert.ToInt32(UInt32 value)
   at Wasmtime.Memory.GetSpan() in C:\Users\VPC\Desktop\WebAssembly-Test\wasmtime-dotnet\src\Memory.cs:line 92
   at Wasmtime.Memory.WriteInt16(Int32 address, Int16 value) in C:\Users\VPC\Desktop\WebAssembly-Test\wasmtime-dotnet\src\Memory.cs:line 228
   at Program.<<Main>$>g__TestBigMemoryCustomWasm|0_3() in C:\Users\VPC\Desktop\WebAssembly-Test\TestWasmtime\Program.cs:line 326

The Memory can currently only be accessed by calling GetSpan() that returns a Span<byte> over the whole memory area. Unfortunately, a Span uses an Int32 length, so it can only have a max length of 2^31-1. Using a memory with 32768 or more pages would exceed that value for a Span<byte> (using a memory with 65536 pages would even exceed the storage of a 32-bit unsigned integer for the length, see also https://github.com/bytecodealliance/wasmtime/pull/3134).

I'm not sure of the best approach to solve this, but I could imagine that we may need new APIs using a long instead of int for addresses (like WriteInt16(long address, short value) etc.), and maybe a Span<byte> GetSpan(long address, int length) that allows to retrieve a Span for the given address and length (when the length doesn't exceed 2^31-1). Additionally, to allow the caller to access the whole memory at once, we might need to allow retrieving a pointer (IntPtr) to the memory start.

(With the Memory64 proposal, a memory can even be larger than 4 GiB. As far is I understand it, it would theoretically support a memory length of 2^64 bytes, which might need to use an ulong as address to access more than 2^63 bytes, but ulong in .NET isn't CLS-compliant, and such a memory size is as of today only a theoretical situation, so I think using a long should be fine.)

What do you think?

Thanks!