burn
burn copied to clipboard
Track disabled HashBrown upgrade
Upgrade of hashbrown from 0.15.5 to 0.16.0 is currently disabled due to version conflicts in downstream dependencies.
Attempting to upgrade hashbrown breaks compatibility due to dependency conflicts, specifically with safetensors.
Upgrade should be enabled once conflicts are resolved in downstream crates.
Relevant PRs:
- Closed PR attempting upgrade: #3800 (Bump hashbrown from 0.15.5 to 0.16.0)
- safetensors change to eliminate hashbrown usage: huggingface/safetensors#656
Upgrade can be re-enabled once the safetensors PR is merged and released.
CC @laggui
@laggui, safetensors team didn't want to change API to use more stable data type BtreeMap instead of HashMap by HashBrown. Instead they suggested to use safetensors exported hashbrown lib: https://github.com/huggingface/safetensors/pull/656#issuecomment-3378585043
They have another PR to upgrade HashBrown lib.
We have two options:
- Use safetensors hashbrown. This affects no-std builds (burn-no-std-tests only) for saving only and minimal impact on build (extra package build time if versions diverge).
- Do not support passing metadata for no-std saving. This breaks functionality however.
Well we also use hashmap as a drop-in replacement for no-std, so I understand their point. It's also a public facing API so that would be a breaking change, and for what 😅
Just had a quick look though, and I think we can already use their re-exported types. So we can replace the usage to:
/// Memory-based store.
pub struct MemoryStore {
data: Option<Arc<Vec<u8>>>,
filter: PathFilter,
#[cfg(feature = "std")]
remapper: KeyRemapper,
metadata: safetensors::lib::HashMap<String, String>, // use the vendored HashMap
validate: bool,
allow_partial: bool,
from_adapter: Box<dyn ModuleAdapter>,
to_adapter: Box<dyn ModuleAdapter>,
}
the contents of the lib module are public.