EliverQ

As far as I understand, when evaluating MTEB in your code, the following lines are used: ```python model = INSTRUCTOR(args.model_name, cache_folder=args.cache_dir) evaluation = MTEB(tasks=[args.task_name], task_langs=["en"]) evaluation.run(model, output_folder=args.output_dir, eval_splits=[args.split], args=args, overwrite_results=True)...

I just use the source code on the Github: https://github.com/HKUNLP/instructor-embedding/blob/main/InstructorEmbedding/instructor.py#L478-L565 ``` def encode(self, sentences, batch_size: int = 32, show_progress_bar: bool = None, output_value: str = 'sentence_embedding', convert_to_numpy: bool = True,...

Sorry, I have tried to install this previously but failed with the message here: ``` ERROR: Could not install packages due to an OSError: [Errno 2] No such file or...

Thanks! I've corrected my evaluation method following your customized mteb package. The performance of replicating INSTRUCTOR have been improved but still lower than yours. Here I still have some detailed...

Thank you! I have other questions. Apologies again for the inconvenience caused by my inquiries. What about INSTRUCTOR-base and INSTRUCTOR-xl? Did you use the exact same experimental setup as INSTRUCTOR-large?...

> Hi, Thanks a lot for your interest in the INSTRUCTOR! > > I trained the INSTRUCTOR model with a single GPU, and the setting is to train 20K steps....

> Hey, we are currently trying to replicate the Instructor model. Issue #14 already asks this, but please report the exact training setup for the models. > > Also, I...

By the way, I think the problem maybe the dtype I use (bf16). But the dtype in your config is fp16 and still doesn't work?

> For the 3B model, since there's no official LLaMA 3B, we defined the model size ourselves and it might not agree with the 3B model sizes in other implementations...

Thank you very much! Perhaps I've been using the code incorrectly all along.