dora chore(deps): update dependencies

This PR contains the following updates:

Package	Change	Type	Update
accelerate	`1.5.1` -> `1.6.0`	project.dependencies	minor
anyhow	`1.0.97` -> `1.0.98`	dependencies	patch
arrow	`54.2.1` -> `54.3.1`	workspace.dependencies	minor
arrow-array	`54.2.1` -> `54.3.1`	workspace.dependencies	minor
arrow-data	`54.3.0` -> `54.3.1`	workspace.dependencies	patch
arrow-schema	`54.3.0` -> `54.3.1`	workspace.dependencies	patch
bytemuck	`1.22.0` -> `1.23.0`	dependencies	minor
chrono	`0.4.39` -> `0.4.41`	dependencies	patch
clap	`4.5.32` -> `4.5.37`	dependencies	patch
ctrlc	`3.4.5` -> `3.4.6`	dependencies	patch
cxx (source)	`1.0.149` -> `1.0.158`	dependencies	patch
cxx-build (source)	`1.0.150` -> `1.0.158`	build-dependencies	patch
dora-rs	`0.3.10` -> `0.3.11`	project.dependencies	patch
dora-rs	`0.3.9` -> `0.3.11`	project.dependencies	patch
env_logger	`0.11.7` -> `0.11.8`	dependencies	patch
grpcio-tools	`1.62.2` -> `1.71.0`	project.dependencies	minor
half	`2.5.0` -> `2.6.0`	dependencies	minor
indexmap	`2.8.0` -> `2.9.0`	dependencies	minor
lebai-sdk	`0.2.23` -> `0.2.26`	project.dependencies	patch
modelscope	`1.24.1` -> `1.25.0`	project.dependencies	minor
numpy (changelog)	`1.24.4` -> `1.26.4`	project.dependencies	minor
once_cell	`1.21.1` -> `1.21.3`	dependencies	patch
open-clip-torch	`2.31.0` -> `2.32.0`	project.dependencies	minor
parquet	`54.2.1` -> `54.3.1`	workspace.dependencies	minor
proc-macro2	`1.0.94` -> `1.0.95`	dependencies	patch
pytest (changelog)	`8.3.4` -> `8.3.5`	dependency-groups	patch
pytest-cov (changelog)	`6.0.0` -> `6.1.1`	dependency-groups	minor
ruff (source, changelog)	`0.9.10` -> `0.11.8`	dependency-groups	minor
ruff (source, changelog)	`0.9.6` -> `0.11.8`	dependency-groups	minor
ruff (source, changelog)	`0.11.4` -> `0.11.8`	dependency-groups	patch
rustdds (source)	`0.11.4` -> `0.11.5`	dependencies	patch
torch	`2.5.1` -> `2.7.0`	project.dependencies	minor
torch	`2.6.0` -> `2.7.0`	project.dependencies	minor
torchvision	`0.21.0` -> `0.22.0`	project.dependencies	minor
transformers	`4.46.3` -> `4.51.3`	project.dependencies	minor
zenoh (source)	`1.3.0` -> `1.3.4`	dependencies	patch

[!WARNING] Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

Release Notes

huggingface/accelerate (accelerate)

`v1.6.0`: : FSDPv2, DeepSpeed TP and XCCL backend support

Compare Source

FSDPv2 support

This release introduces the support for FSDPv2 thanks to @S1ro1.

If you are using python code, you need to set fsdp_version=2 in FullyShardedDataParallelPlugin:

from accelerate import FullyShardedDataParallelPlugin, Accelerator

fsdp_plugin = FullyShardedDataParallelPlugin(
    fsdp_version=2

### other options...
)
accelerator = Accelerator(fsdp_plugin=fsdp_plugin)

If want to convert a YAML config that contains the FSDPv1 config to FSDPv2 one , use our conversion tool:

accelerate to-fsdp2 --config_file config.yaml --output_file new_config.yaml`

To learn more about the difference between FSDPv1 and FSDPv2, read the following documentation.

DeepSpeed TP support

We have added initial support for DeepSpeed + TP. Not many changes were required as the DeepSpeed APIs was already compatible. We only needed to make sure that the dataloader was compatible with TP and that we were able to save the TP weights. Thanks @inkcherry for the work ! https://github.com/huggingface/accelerate/pull/3390.

To use TP with deepspeed, you need to update the setting in the deepspeed config file by including tensor_parallel key:

    ....
    "tensor_parallel":{
      "autotp_size": ${autotp_size}
    },
   ...

More details in this deepspeed PR.

Support for XCCL distributed backend

We've added support for XCCL which is an Intel distributed backend which can be used with XPU devices. More details in this torch PR. Thanks @dvrogozh for the integration !

What's Changed

Add log_artifact, log_artifacts and log_figure capabilities to the MLflowTracker. by @luiz0992 in https://github.com/huggingface/accelerate/pull/3419
tensor parallel dataloder for deepspeed accelerator by @inkcherry in https://github.com/huggingface/accelerate/pull/3390
Fix prod issues by @muellerzr in https://github.com/huggingface/accelerate/pull/3441
Fix attribute issue with deepspeed tp by @SunMarc in https://github.com/huggingface/accelerate/pull/3443
Fixed typo in the multi node FSDP slurm example script by @JacobB33 in https://github.com/huggingface/accelerate/pull/3447
feat: Add no_ssh and slurm multinode launcher options for deepspeed by @hsmallbone in https://github.com/huggingface/accelerate/pull/3329
Fixup ao module filter func by @muellerzr in https://github.com/huggingface/accelerate/pull/3450
remove device index workaround on xpu since xpu supports integer device index as cuda now by @yao-matrix in https://github.com/huggingface/accelerate/pull/3448
enable 2 UT cases on XPU by @yao-matrix in https://github.com/huggingface/accelerate/pull/3445
Fix AMD GPU support with should_reduce_batch_size() by @cameronshinn in https://github.com/huggingface/accelerate/pull/3405
Fix device KeyError in tied_params_map by @dvrogozh in https://github.com/huggingface/accelerate/pull/3403
Initial FSDP2 support by @S1ro1 in https://github.com/huggingface/accelerate/pull/3394
Fix: clip grad norm in fsdp2 by @S1ro1 in https://github.com/huggingface/accelerate/pull/3465
Update @ by @muellerzr in https://github.com/huggingface/accelerate/pull/3466
Fix seeding of new generator for multi GPU by @albertcthomas in https://github.com/huggingface/accelerate/pull/3459
Fix get_balanced_memory for MPS by @booxter in https://github.com/huggingface/accelerate/pull/3464
Update CometMLTracker to allow re-using experiment by @Lothiraldan in https://github.com/huggingface/accelerate/pull/3328
Apply ruff py39 fixes by @cyyever in https://github.com/huggingface/accelerate/pull/3461
xpu: enable xccl distributed backend by @dvrogozh in https://github.com/huggingface/accelerate/pull/3401
Update ruff target-version to py39 and apply more fixes by @cyyever in https://github.com/huggingface/accelerate/pull/3470
[MLU] fix deepspeed dependency by @huismiling in https://github.com/huggingface/accelerate/pull/3472
remove use_xpu to fix ut issues, we don't need this since XPU is OOB … by @yao-matrix in https://github.com/huggingface/accelerate/pull/3460
Bump ruff to 0.11.2 by @cyyever in https://github.com/huggingface/accelerate/pull/3471

New Contributors

@luiz0992 made their first contribution in https://github.com/huggingface/accelerate/pull/3419
@inkcherry made their first contribution in https://github.com/huggingface/accelerate/pull/3390
@JacobB33 made their first contribution in https://github.com/huggingface/accelerate/pull/3447
@hsmallbone made their first contribution in https://github.com/huggingface/accelerate/pull/3329
@yao-matrix made their first contribution in https://github.com/huggingface/accelerate/pull/3448
@cameronshinn made their first contribution in https://github.com/huggingface/accelerate/pull/3405
@S1ro1 made their first contribution in https://github.com/huggingface/accelerate/pull/3394
@albertcthomas made their first contribution in https://github.com/huggingface/accelerate/pull/3459
@booxter made their first contribution in https://github.com/huggingface/accelerate/pull/3464
@Lothiraldan made their first contribution in https://github.com/huggingface/accelerate/pull/3328
@cyyever made their first contribution in https://github.com/huggingface/accelerate/pull/3461

Full Changelog: https://github.com/huggingface/accelerate/compare/v1.5.2...v1.6.0

`v1.5.2`: Patch: v1.5.2

Compare Source

Bug Fixes:

Fixed an issue with torch.get_default_device() requiring a higher version than what we support
Fixed a broken pytest import in prod

Full Changelog: https://github.com/huggingface/accelerate/compare/v1.5.0...v1.5.2

dtolnay/anyhow (anyhow)

`v1.0.98`

Compare Source

Add self.into_boxed_dyn_error() and self.reallocate_into_boxed_dyn_error_without_backtrace() methods for anyhow::Error (#415)

apache/arrow-rs (arrow)

`v54.3.1`: arrow 54.3.1

Compare Source

Changelog

54.3.1 (2025-03-26)

Full Changelog

Fixed bugs:

Round trip encoding of list of fixed list fails when offset is not zero #7315

Merged pull requests:

Add missing type annotation #7326 [parquet] (mbrobbel)
bugfix: correct offsets when serializing a list of fixed sized list and non-zero start offset #7318 [arrow] (timsaucer)

* This Changelog was automatically generated by github_changelog_generator

`v54.3.0`: arrow 54.3.0

Compare Source

Changelog

54.3.0 (2025-03-17)

Full Changelog

Implemented enhancements:

Using column chunk offset index in InMemoryRowGroup::fetch #7300
Support reading parquet with modular encryption #7296 [parquet]
Add example for how to read/write encrypted parquet files #7281 [parquet]
Have writer return parsed ParquetMetadata #7254 [parquet]
feat: Support Utf8View in JSON reader #7244 [arrow]
StructBuilder should provide a way to get a &dyn ArrayBuilder of a field builder #7193 [arrow]
Support div_wrapping/rem_wrapping for numeric arithmetic kernels #7158 [arrow]
Improve RleDecoder performance #7195 [parquet] (Dandandan)
Improve arrow-json deserialization performance by 30% #7157 [arrow] (mwylde)
Add with_skip_validation flag to IPC StreamReader, FileReader and FileDecoder #7120 [arrow] (alamb)

Fixed bugs:

Archery integration CI test is failing on main: error: package half v2.5.0 cannot be built because it requires rustc 1.81 or newer, while the currently active rustc version is 1.77.2 #7291
MSRV CI check is failing on main #7289
Incorrect IPC schema encoding for multiple dictionaries #7058 [arrow] [arrow-flight]

Documentation updates:

Add example for how to read encrypted parquet files #7283 [parquet] (rok)
Update the relative path of the test data in docs #7221 (Ziy1-Tan)
Minor: fix doc and remove unused code #7194 [arrow] (lewiszlw)
doc: modify wrong comment #7190 [arrow] (YichiZhang0613)
doc: fix IPC file reader/writer docs #7178 [arrow] (Jefffrey)

Merged pull requests:

chore: require ffi feature in arrow-schema benchmark #7298 [arrow] (ethe)
Fix archery integration test #7292 (alamb)
Minor: run test_decimal_list again #7282 [parquet] (alamb)
Move Parquet encryption tests into the arrow_reader integration tests #7279 [parquet] (adamreeve)
Include license and notice files in published crates, part 2 #7275 [arrow] (ankane)
feat: Support Utf8View in JSON reader #7263 [arrow] (zhuqi-lucas)
feat: use force_validate feature flag when creating an arrays #7241 [arrow] (rluvaton)
fix: take on empty struct array returns empty array #7224 [arrow] (westonpace)
fix: correct bloom_filter_position description #7223 [parquet] (romanz)
Minor: Move make_builder into mod.rs #7218 (lewiszlw)
Expose field_builders in StructBuilder #7217 [arrow] (lewiszlw)
Minor: Fix json StructMode docs links #7215 [arrow] (gstvg)
[main] Bump arrow version to 54.2.1 (#7207) #7212 (alamb)
feat: add downcast_integer_array macro helper #7211 [arrow] (rluvaton)
Remove zstd pin #7199 [parquet] (tustvold)
fix: Use chrono's quarter() to avoid conflict #7198 [arrow] (yutannihilation)
Fix some Clippy 1.85 warnings #7167 [parquet] [arrow] (mbrobbel)
feat: add to concat different data types error message the data types #7166 [arrow] (rluvaton)
Add Week ISO, Year ISO computation #7163 [arrow] (kosiew)
fix: create_random_batch fails with timestamp types having a timezone #7162 [arrow] (niebayes)
Avoid overflow of remainder #7159 [arrow] (wForget)
fix: Data type inference for NaN, inf and -inf in csv files #7150 [arrow] (Mottl)
Preserve null dictionary values in interleave and concat kernels #7144 [arrow] (kawadakk)
Support casting Date to a time zone-specific timestamp #7141 [arrow] (friendlymatthew)
Minor: Add doctest to ArrayDataBuilder::build_unchecked #7139 [arrow] (gstvg)
arrow-ord: add support for nested types to partition #7131 [arrow] (asubiotto)
Update prost-build requirement from =0.13.4 to =0.13.5 #7127 [arrow] [arrow-flight] (dependabot[bot])
Avoid use of flatbuffers::size_prefixed_root, fix validation error in arrow-flight #7109 [arrow] [arrow-flight] (bkietz)
Optimise decimal casting for infallible conversions #7021 [arrow] (aweltsch)