scuffle icon indicating copy to clipboard operation
scuffle copied to clipboard
verified publisher icon ScuffleTV

Add deserializer for amf0 value and fix amf0 visitor fns

Open philipch07 opened this issue 8 months ago • 5 comments

The deserializer is a qol feature for people using amf0.

The current amf0 visitor functions do not properly call the corresponding visit functions, for example, an i8 currently calls self.deserialize_i64(visitor) (incorrect) instead of self.deserialize_i8(visitor) (correct).

For more on the above topic, see the Notion doc. Just a heads up that it's also missing a rule which is mentioned in Troy's other https://github.com/ScuffleCloud/scuffle/pull/441#discussion_r2116965102

Additionally this PR aims to address the following from Troy's comment (see below):

We should make [the Amf0Object] a wrapper instead of a type def.

So right now this is assigned to be a BTreeMap, this is less than ideal because in the future we might want to preserve order strictly (ie an IndexMap) and that would be a breaking change. Instead we should make a new structure called Amf0Object and then internally we have a BTreeMap and then we provide a bunch of helper functions reexporting the BTreeMap's behaviours (len, is_empty, iter, into_iter, get, get_mut, remove, insert, from_iter, etc)

philipch07 avatar Apr 27 '25 22:04 philipch07

Deploying scuffle-docusaurus-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 51e201e
Status: ✅  Deploy successful!
Preview URL: https://42898d9c.scuffle-docusaurus-docs.pages.dev
Branch Preview URL: https://pr-441.scuffle-docusaurus-docs.pages.dev

View logs

cloudflare-workers-and-pages[bot] avatar Apr 28 '25 00:04 cloudflare-workers-and-pages[bot]

Deploying scuffle-docrs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 51e201e
Status: ✅  Deploy successful!
Preview URL: https://4c3f1547.scuffle-docrs.pages.dev
Branch Preview URL: https://pr-441.scuffle-docrs.pages.dev

View logs

cloudflare-workers-and-pages[bot] avatar Apr 28 '25 02:04 cloudflare-workers-and-pages[bot]

:x: 4 Tests Failed:

Tests completed Failed Passed Skipped
1076 4 1072 0
View the top 3 failed test(s) by shortest run time 
scuffle-amf0::value::tests::roundtrip_enum_unit
Stack Traces | 0.003s run time 
thread 'value::tests::roundtrip_enum_unit' panicked at .../amf0/src/value.rs:981:94:
called `Result::unwrap()` on an `Err` value: Custom("invalid type: string \"Unit\", expected enum TestEnum")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
scuffle-amf0::value::tests::roundtrip_enum_map
Stack Traces | 0.014s run time 
thread 'value::tests::roundtrip_enum_map' panicked at .../amf0/src/value.rs:1067:74:
called `Result::unwrap()` on an `Err` value: Io(Custom { kind: UnexpectedEof, error: "Not enough data" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
scuffle-amf0::value::tests::roundtrip_enum_struct
Stack Traces | 0.014s run time 
thread 'value::tests::roundtrip_enum_struct' panicked at .../amf0/src/value.rs:1023:94:
called `Result::unwrap()` on an `Err` value: Custom("invalid type: string \"Struct\", expected enum TestEnum")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

To view more test analytics, go to the Test Analytics Dashboard 📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

codecov[bot] avatar Apr 28 '25 02:04 codecov[bot]

✅ Release Checks Passed

⭐ Package Changes

  • scuffle-amf0: ⚠️ Breaking Change

    Cargo semver-checks details 
    --- failure method_requires_different_generic_type_params: method now requires a different number of generic type parameters ---

Description:
A method now requires a different number of generic type parameters than it used to. Uses of this method that supplied the previous number of generic types will be broken.
        ref: https://doc.rust-lang.org/reference/items/generics.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.41.0/src/lints/method_requires_different_generic_type_params.ron

Failed in:
  scuffle_amf0::encoder::Amf0Encoder::encode_array takes 2 generic types instead of 0, in ./crates/amf0/src/encoder.rs:87
  scuffle_amf0::Amf0Encoder::encode_array takes 2 generic types instead of 0, in ./crates/amf0/src/encoder.rs:87

  • scuffle-bytes-util: 🛠️ Compatiable Change

  • 🔒 xtask: 🛠️ Compatiable Change

cargo xtask release check 
    Updating crates.io index
 Downloading crates ...
  Downloaded block-buffer v0.10.4
  Downloaded anstyle-parse v0.2.7
  Downloaded anstyle v1.0.11
  Downloaded camino v1.1.10
  Downloaded cargo-platform v0.2.0
  Downloaded anyhow v1.0.98
  Downloaded anstream v0.6.19
  Downloaded aho-corasick v1.1.3
  Downloaded autocfg v1.5.0
  Downloaded anstyle-query v1.1.3
  Downloaded writeable v0.6.1
  Downloaded utf8parse v0.2.2
  Downloaded utf8_iter v1.0.4
  Downloaded unicode-xid v0.2.6
  Downloaded zerofrom v0.1.6
  Downloaded zerovec-derive v0.11.1
  Downloaded zerofrom-derive v0.1.6
  Downloaded yoke v0.8.0
  Downloaded url v2.5.4
  Downloaded zerotrie v0.2.2
  Downloaded indent_write v2.2.0
  Downloaded zerovec v0.11.2
  Downloaded litemap v0.8.0
  Downloaded idna v1.0.3
  Downloaded matchers v0.1.0
  Downloaded minijinja v2.10.2
  Downloaded heck v0.5.0
  Downloaded winnow v0.7.11
  Downloaded nu-ansi-term v0.46.0
  Downloaded log v0.4.27
  Downloaded serde-untagged v0.1.7
  Downloaded ordered-float v2.10.1
  Downloaded is_terminal_polyfill v1.70.1
  Downloaded icu_provider v2.0.0
  Downloaded num-traits v0.2.19
  Downloaded num_cpus v1.17.0
  Downloaded stable_deref_trait v1.2.0
  Downloaded indexmap v2.9.0
  Downloaded sha2 v0.10.9
  Downloaded semver v1.0.26
  Downloaded typeid v1.0.3
  Downloaded thread_local v1.1.9
  Downloaded toml_datetime v0.6.11
  Downloaded tracing-core v0.1.34
  Downloaded tracing-log v0.2.0
  Downloaded crypto-common v0.1.6
  Downloaded serde-value v0.7.0
  Downloaded icu_normalizer v2.0.0
  Downloaded strsim v0.11.1
  Downloaded serde v1.0.219
  Downloaded thiserror-impl v2.0.12
  Downloaded toml_write v0.1.2
  Downloaded toml v0.8.23
  Downloaded sharded-slab v0.1.7
  Downloaded tracing v0.1.41
  Downloaded regex-automata v0.1.10
  Downloaded regex-syntax v0.6.29
  Downloaded icu_properties_data v2.0.1
  Downloaded hashbrown v0.15.4
  Downloaded clap_builder v4.5.40
  Downloaded tracing-subscriber v0.3.19
  Downloaded libc v0.2.174
  Downloaded chrono v0.4.41
  Downloaded syn v2.0.104
  Downloaded icu_normalizer_data v2.0.0
  Downloaded regex-syntax v0.8.5
  Downloaded icu_locale_core v2.0.0
  Downloaded icu_collections v2.0.0
  Downloaded erased-serde v0.4.6
  Downloaded clap v4.5.40
  Downloaded proc-macro2 v1.0.95
  Downloaded icu_properties v2.0.1
  Downloaded cpufeatures v0.2.17
  Downloaded clap_derive v4.5.40
  Downloaded serde_derive v1.0.219
  Downloaded iana-time-zone v0.1.63
  Downloaded cargo-util-schemas v0.2.0
  Downloaded hex v0.4.3
  Downloaded generic-array v0.14.7
  Downloaded form_urlencoded v1.2.1
  Downloaded displaydoc v0.2.5
  Downloaded clap_lex v0.7.5
  Downloaded cfg-if v1.0.1
  Downloaded regex-automata v0.4.9
  Downloaded cargo_metadata v0.20.0
  Downloaded serde_spanned v0.6.9
  Downloaded fmtools v0.1.2
  Downloaded ryu v1.0.20
  Downloaded quote v1.0.40
  Downloaded home v0.5.11
  Downloaded tracing-attributes v0.1.30
  Downloaded toml_edit v0.22.27
  Downloaded potential_utf v0.1.2
  Downloaded equivalent v1.0.2
  Downloaded digest v0.10.7
  Downloaded thiserror-impl v1.0.69
  Downloaded thiserror v1.0.69
  Downloaded smallvec v1.15.1
  Downloaded serde_json v1.0.140
  Downloaded regex v1.11.1
  Downloaded percent-encoding v2.3.1
  Downloaded tinystr v0.8.1
  Downloaded thiserror v2.0.12
  Downloaded synstructure v0.13.2
  Downloaded overload v0.1.1
  Downloaded colorchoice v1.0.4
  Downloaded once_cell v1.21.3
  Downloaded memchr v2.7.5
  Downloaded pin-project-lite v0.2.16
  Downloaded lazy_static v1.5.0
  Downloaded itoa v1.0.15
  Downloaded unicode-ident v1.0.18
  Downloaded typenum v1.18.0
  Downloaded yoke-derive v0.8.0
  Downloaded idna_adapter v1.2.1
  Downloaded version_check v0.9.5
   Compiling proc-macro2 v1.0.95
   Compiling unicode-ident v1.0.18
   Compiling serde v1.0.219
   Compiling stable_deref_trait v1.2.0
   Compiling litemap v0.8.0
   Compiling writeable v0.6.1
   Compiling icu_properties_data v2.0.1
   Compiling icu_normalizer_data v2.0.0
   Compiling autocfg v1.5.0
   Compiling version_check v0.9.5
   Compiling smallvec v1.15.1
   Compiling typenum v1.18.0
   Compiling typeid v1.0.3
   Compiling hashbrown v0.15.4
   Compiling equivalent v1.0.2
   Compiling generic-array v0.14.7
   Compiling thiserror v1.0.69
   Compiling utf8parse v0.2.2
   Compiling num-traits v0.2.19
   Compiling toml_write v0.1.2
   Compiling utf8_iter v1.0.4
   Compiling winnow v0.7.11
   Compiling percent-encoding v2.3.1
   Compiling once_cell v1.21.3
   Compiling semver v1.0.26
   Compiling quote v1.0.40
   Compiling tracing-core v0.1.34
   Compiling syn v2.0.104
   Compiling indexmap v2.9.0
   Compiling form_urlencoded v1.2.1
   Compiling anstyle-parse v0.2.7
   Compiling is_terminal_polyfill v1.70.1
   Compiling anstyle v1.0.11
   Compiling thiserror v2.0.12
   Compiling colorchoice v1.0.4
   Compiling cfg-if v1.0.1
   Compiling anstyle-query v1.1.3
   Compiling ordered-float v2.10.1
   Compiling libc v0.2.174
   Compiling regex-syntax v0.6.29
   Compiling serde_json v1.0.140
   Compiling camino v1.1.10
   Compiling regex-syntax v0.8.5
   Compiling block-buffer v0.10.4
   Compiling crypto-common v0.1.6
   Compiling anstream v0.6.19
   Compiling strsim v0.11.1
   Compiling lazy_static v1.5.0
   Compiling heck v0.5.0
   Compiling log v0.4.27
   Compiling itoa v1.0.15
   Compiling ryu v1.0.20
   Compiling anyhow v1.0.98
   Compiling memchr v2.7.5
   Compiling clap_lex v0.7.5
   Compiling overload v0.1.1
   Compiling pin-project-lite v0.2.16
   Compiling unicode-xid v0.2.6
   Compiling clap_builder v4.5.40
   Compiling nu-ansi-term v0.46.0
   Compiling tracing-log v0.2.0
   Compiling sharded-slab v0.1.7
   Compiling digest v0.10.7
   Compiling thread_local v1.1.9
   Compiling cpufeatures v0.2.17
   Compiling iana-time-zone v0.1.63
   Compiling sha2 v0.10.9
   Compiling regex-automata v0.4.9
   Compiling regex-automata v0.1.10
   Compiling num_cpus v1.17.0
   Compiling chrono v0.4.41
   Compiling matchers v0.1.0
   Compiling home v0.5.11
   Compiling indent_write v2.2.0
   Compiling hex v0.4.3
   Compiling fmtools v0.1.2
   Compiling regex v1.11.1
   Compiling synstructure v0.13.2
   Compiling zerofrom-derive v0.1.6
   Compiling serde_derive v1.0.219
   Compiling yoke-derive v0.8.0
   Compiling zerovec-derive v0.11.1
   Compiling displaydoc v0.2.5
   Compiling thiserror-impl v1.0.69
   Compiling thiserror-impl v2.0.12
   Compiling tracing-attributes v0.1.30
   Compiling clap_derive v4.5.40
   Compiling zerofrom v0.1.6
   Compiling tracing v0.1.41
   Compiling yoke v0.8.0
   Compiling tracing-subscriber v0.3.19
   Compiling zerovec v0.11.2
   Compiling zerotrie v0.2.2
   Compiling clap v4.5.40
   Compiling tinystr v0.8.1
   Compiling potential_utf v0.1.2
   Compiling icu_collections v2.0.0
   Compiling icu_locale_core v2.0.0
   Compiling icu_provider v2.0.0
   Compiling icu_properties v2.0.1
   Compiling icu_normalizer v2.0.0
   Compiling idna_adapter v1.2.1
   Compiling idna v1.0.3
   Compiling url v2.5.4
   Compiling toml_datetime v0.6.11
   Compiling serde_spanned v0.6.9
   Compiling erased-serde v0.4.6
   Compiling serde-value v0.7.0
   Compiling cargo-platform v0.2.0
   Compiling toml_edit v0.22.27
   Compiling minijinja v2.10.2
   Compiling serde-untagged v0.1.7
   Compiling toml v0.8.23
   Compiling cargo-util-schemas v0.2.0
   Compiling cargo_metadata v0.20.0
   Compiling xtask v0.0.0 (/home/runner/work/scuffle/scuffle/dev-tools/xtask)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 10.77s
     Running `target/xtask/debug/xtask release check --exit-status --pr-number 441`
2025-06-22T16:19:50.274779Z  INFO crates_io_versions{package=openapiv3_1}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/op/en/openapiv3_1
2025-06-22T16:19:50.274781Z  INFO crates_io_versions{package=nutype-enum}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/nu/ty/nutype-enum
2025-06-22T16:19:50.274799Z  INFO crates_io_versions{package=scuffle-av1}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-av1
2025-06-22T16:19:50.274813Z  INFO crates_io_versions{package=postcompile}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/po/st/postcompile
2025-06-22T16:19:50.274797Z  INFO crates_io_versions{package=scuffle-aac}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-aac
2025-06-22T16:19:50.274843Z  INFO crates_io_versions{package=scuffle-batching}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-batching
2025-06-22T16:19:50.274861Z  INFO crates_io_versions{package=scuffle-amf0}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-amf0
2025-06-22T16:19:50.274870Z  INFO crates_io_versions{package=scuffle-bootstrap}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-bootstrap
2025-06-22T16:19:50.512451Z  INFO crates_io_versions{package=scuffle-changelog}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-changelog
2025-06-22T16:19:50.790930Z  INFO crates_io_versions{package=scuffle-flv}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-flv
2025-06-22T16:19:50.792062Z  INFO crates_io_versions{package=scuffle-bootstrap-derive}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-bootstrap-derive
2025-06-22T16:19:50.797790Z  INFO crates_io_versions{package=scuffle-bootstrap-telemetry}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-bootstrap-telemetry
2025-06-22T16:19:50.806874Z  INFO crates_io_versions{package=scuffle-transmuxer}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-transmuxer
2025-06-22T16:19:50.815215Z  INFO crates_io_versions{package=scuffle-h265}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-h265
2025-06-22T16:19:50.819128Z  INFO crates_io_versions{package=scuffle-expgolomb}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-expgolomb
2025-06-22T16:19:50.826312Z  INFO crates_io_versions{package=scuffle-signal}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-signal
2025-06-22T16:19:51.013470Z  INFO crates_io_versions{package=scuffle-pprof}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-pprof
2025-06-22T16:19:51.022058Z  INFO crates_io_versions{package=tinc-cel}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/ti/nc/tinc-cel
2025-06-22T16:19:51.046129Z  INFO crates_io_versions{package=scuffle-rtmp}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-rtmp
2025-06-22T16:19:51.314568Z  INFO crates_io_versions{package=scuffle-metrics-derive}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-metrics-derive
2025-06-22T16:19:51.319507Z  INFO crates_io_versions{package=scuffle-mp4}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-mp4
2025-06-22T16:19:51.324519Z  INFO crates_io_versions{package=scuffle-settings}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-settings
2025-06-22T16:19:51.333658Z  INFO crates_io_versions{package=tinc}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/ti/nc/tinc
2025-06-22T16:19:51.343250Z  INFO crates_io_versions{package=tinc-pb-prost}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/ti/nc/tinc-pb-prost
2025-06-22T16:19:51.532797Z  INFO crates_io_versions{package=scuffle-bytes-util}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-bytes-util
2025-06-22T16:19:51.550472Z  INFO crates_io_versions{package=scuffle-ffmpeg}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-ffmpeg
2025-06-22T16:19:51.558064Z  INFO crates_io_versions{package=scuffle-future-ext}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-future-ext
2025-06-22T16:19:51.834722Z  INFO crates_io_versions{package=scuffle-http}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-http
2025-06-22T16:19:51.837974Z  INFO crates_io_versions{package=scuffle-h264}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-h264
2025-06-22T16:19:51.843918Z  INFO crates_io_versions{package=tinc-build}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/ti/nc/tinc-build
2025-06-22T16:19:51.856373Z  INFO crates_io_versions{package=tinc-derive}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/ti/nc/tinc-derive
2025-06-22T16:19:51.880570Z  INFO crates_io_versions{package=scuffle-metrics}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-metrics
2025-06-22T16:19:52.064861Z  INFO crates_io_versions{package=scuffle-context}: xtask::cmd::release::utils: checking on crates.io url=https://index.crates.io/sc/uf/scuffle-context
2025-06-22T16:19:52.585209Z  INFO check{package=scuffle-amf0}:download_crate{name=scuffle-amf0 version=0.2.4}: xtask::cmd::release::utils: fetching from crates.io url=https://static.crates.io/crates/scuffle-amf0/scuffle-amf0-0.2.4.crate
2025-06-22T16:19:52.589520Z  INFO check{package=scuffle-bytes-util}:download_crate{name=scuffle-bytes-util version=0.1.4}: xtask::cmd::release::utils: fetching from crates.io url=https://static.crates.io/crates/scuffle-bytes-util/scuffle-bytes-util-0.1.4.crate
2025-06-22T16:19:52.633349Z  INFO xtask::cmd::release::check: running cargo-semver-checks
2025-06-22T16:19:59.079892Z  INFO xtask::cmd::release::check: running min versions check
2025-06-22T16:20:08.664111Z  INFO xtask::cmd::release::check: complete

github-actions[bot] avatar May 26 '25 18:05 github-actions[bot]

@philipch07 i pushed a commit showing sort of what i was looking for. What is currently outstanding on this PR is testing forward / backward ser-de impls.

Meaning that everytime we serialize should be able to be deserialized back.

So the goal is to now:

test serde-back-and-forth for all types

Sort of like this flow

let rust_type = something;
let amf0_bytes = rust_type.encode();
let rust_type2 = amf0_bytes.decode();
assert_eq(rust_type, rust_type2);

let amf0_value= amf0_bytes.decode();
let rust_type3 = amf0_value.decode();
assert_eq(rust_type, rust_type3);

TroyKomodo avatar Jun 09 '25 12:06 TroyKomodo