burn icon indicating copy to clipboard operation
burn copied to clipboard

Track disabled HashBrown upgrade

Open antimora opened this issue 2 months ago • 3 comments

Upgrade of hashbrown from 0.15.5 to 0.16.0 is currently disabled due to version conflicts in downstream dependencies.

Attempting to upgrade hashbrown breaks compatibility due to dependency conflicts, specifically with safetensors.

Upgrade should be enabled once conflicts are resolved in downstream crates.

Relevant PRs:

Upgrade can be re-enabled once the safetensors PR is merged and released.

antimora avatar Oct 02 '25 21:10 antimora

CC @laggui

antimora avatar Oct 02 '25 21:10 antimora

@laggui, safetensors team didn't want to change API to use more stable data type BtreeMap instead of HashMap by HashBrown. Instead they suggested to use safetensors exported hashbrown lib: https://github.com/huggingface/safetensors/pull/656#issuecomment-3378585043

They have another PR to upgrade HashBrown lib.

We have two options:

  1. Use safetensors hashbrown. This affects no-std builds (burn-no-std-tests only) for saving only and minimal impact on build (extra package build time if versions diverge).
  2. Do not support passing metadata for no-std saving. This breaks functionality however.

antimora avatar Oct 14 '25 16:10 antimora

Well we also use hashmap as a drop-in replacement for no-std, so I understand their point. It's also a public facing API so that would be a breaking change, and for what 😅

Just had a quick look though, and I think we can already use their re-exported types. So we can replace the usage to:

/// Memory-based store.
pub struct MemoryStore {
    data: Option<Arc<Vec<u8>>>,
    filter: PathFilter,
    #[cfg(feature = "std")]
    remapper: KeyRemapper,
    metadata: safetensors::lib::HashMap<String, String>, // use the vendored HashMap
    validate: bool,
    allow_partial: bool,
    from_adapter: Box<dyn ModuleAdapter>,
    to_adapter: Box<dyn ModuleAdapter>,
}

the contents of the lib module are public.

laggui avatar Oct 15 '25 12:10 laggui