fix: check filetype in `path_open`

[rvolosatovs]

Using the directory open flag to determine whether the entity being opened is a directory or a file is undefined behavior, since that means that the only way to acquire a usable directory is to specify the flag.

Instead of relying on the open flag value to determine the filetype, issue get_path_filestat first to determine it and then perform appropriate operation depending on the value.

~Note that this also slightly changes the behavior, where create open flag is not dropped anymore, but rather passed through to the open_dir call.~

@pchickey seems to be the last person making any significant changes in this method, so assigning him for review

[github-actions[bot]]

Subscribe to Label Action

cc @kubkon

This issue or pull request has been labeled: "wasi"

Thus the following users have been cc'd because of the following labels:

• kubkon: wasi

To subscribe or unsubscribe from this label, edit the .github/subscribe-to-label.json configuration file.

Learn more.

[npmccallum]

@pchickey @sunfishcode We are blocked on this and would appreciate a prompt review. Thanks!

[rvolosatovs]

I've rebased the PR on latest main and added a commit, which extends the existing readdir test case and can be used to reproduce the error on latest main https://github.com/bytecodealliance/wasmtime/pull/4947/commits/fe33e2990bae149042361efcbb109541b734d4d8

This is the failure on latest main (fe33e2990bae149042361efcbb109541b734d4d8 main with the testing commit applied):

Error: error while testing Wasm module 'fd_readdir'

Caused by:
    wasm trap: wasm `unreachable` instruction executed
    wasm backtrace:
        0: 0x928b - <unknown>!__rust_start_panic
        1: 0x91d2 - <unknown>!rust_panic
        2: 0x91a1 - <unknown>!std::panicking::rust_panic_with_hook::hc938c0552bc648ce
        3: 0x8383 - <unknown>!std::panicking::begin_panic_handler::{{closure}}::h4d01c9a24b253a57
        4: 0x82b0 - <unknown>!std::sys_common::backtrace::__rust_end_short_backtrace::hfd66980d7bb1538d
        5: 0x8a43 - <unknown>!rust_begin_unwind
        6: 0xe71c - <unknown>!core::panicking::panic_fmt::h8d02db942d8b9e8f
        7: 0xfee7 - <unknown>!core::result::unwrap_failed::h806695779fe36d9c
        8: 0x1fa7 - <unknown>!fd_readdir::assert_empty_dir::h0e0aed0574ce8ed0
        9: 0x28e3 - <unknown>!fd_readdir::main::h341e702fce5fa376
       10:  0xa24 - <unknown>!std::sys_common::backtrace::__rust_begin_short_backtrace::h9de52a2ebd9540fa
       11: 0x1a6e - <unknown>!std::rt::lang_start::{{closure}}::h74c1c1c4b8e1fa2a
       12: 0x5f8f - <unknown>!std::rt::lang_start_internal::h5e0caf35c22fb11b
       13: 0x3653 - <unknown>!__original_main
       14:  0x56d - <unknown>!_start
       15: 0x1367f - <unknown>!_start.command_export
    note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable to may show more debugging information
    
test wasi_cap_std_sync::fd_readdir ... FAILED

[npmccallum]

@pchickey I disagree strongly that downcasting should ever be used in wasi-common. In our wasi implementation, we are regularly working around downcasting just to make things work. Adopting downcasting in the first place was a huge mistake.

[pchickey]

@npmccallum OK, send a PR to fix it then

[npmccallum]

@pchickey I discussed this and other issues with @sunfishcode and he recommended that I not try to patch the existing implementation and wait until preview2. Should I file an issue requesting the removal of downcasting under preview2?