chatsci

How to get the whisper, clip, and llama model used by macaw?

5

I used the following code to get the pretrained models: from transformers import CLIPModel, LlamaModel clip_model = CLIPModel.from_pretrained("openai/clip-vit-base-patch16") from transformers import WhisperForConditionalGeneration whisper_model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-base") llama7b_model = LlamaModel.from_pretrained("decapoda-research/llama-7b-hf") clip_model.save_pretrained('trained_models/clip_model/') whisper_model.save_pretrained('trained_models/whisper_model/')...

Data filtering step

1

When processing the dataset, there is a filter criteria: if 'caption' in e['instruction'] or 'caption' in e['response'] or ' no ' in e['response'] or 'not' in e['response']: continue Why we...