tensor_parallel icon indicating copy to clipboard operation
tensor_parallel copied to clipboard

tensor_parallel int4 LLM is not working since release v2.0.0

Open ReinForce-II opened this issue 1 year ago • 0 comments

It works fine on v1.3.2, however

RuntimeError: Trying to shard a model containing 'meta' parameters. Please set `sharded=False` during model creation and call `.apply_sharding()` only after dispatch

occurres when calling

tp.TensorParallelPreTrainedModel(...)

on v2.0.0

ReinForce-II avatar Jan 02 '24 08:01 ReinForce-II