How to use api for t5 and example dataset?

[djaffer]

trafficstars

Is the api inference available for t5.

Any example available? Also what's the format of data.

[merrymercy]

@DachengLi1

[DachengLi1]

You can use the following command to fine-tune a FastChat-T5. The data format is the same as Vicuna one. torchrun --nproc_per_node=4 --master_port=9778 fastchat/train/train_flant5.py
--model_name_or_path google/flan-t5-xl
--data_path playground/data/dummy.json
--bf16 True
--output_dir ./checkpoints_flant5_3b
--num_train_epochs 3
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--gradient_accumulation_steps 4
--evaluation_strategy "no"
--save_strategy "steps"
--save_steps 300
--save_total_limit 1
--learning_rate 2e-5
--weight_decay 0.
--warmup_ratio 0.03
--lr_scheduler_type "cosine"
--logging_steps 1
--fsdp "full_shard auto_wrap"
--fsdp_transformer_layer_cls_to_wrap T5Block
--tf32 True
--model_max_length 2048
--preprocessed_path ./preprocessed_data/processed.json
--gradient_checkpointing True

[DachengLi1]

@merrymercy Do we have plan to include T5 API?

[merrymercy]

Do you mean the huggingface API or OpenAI-compatible API?

both of them are supported with the latest main branch. See the help https://github.com/lm-sys/FastChat#api

[DachengLi1]

Closing this as I should have provided necessary information. Feel free to reopen.