llm-foundry icon indicating copy to clipboard operation
llm-foundry copied to clipboard

Published 20 hours ago • verified publisher icon mosaicml

Add FreebaseQA to tasks and gauntlet

Open moeiniamir opened this issue 10 months ago • 2 comments

Added FreebaseQA which is used in several recent papers 1 2 3. Tested on Vicuna-7B and obtained similar results to 3. Feel free to cherry-pick if we only need the JSON file itself.

moeiniamir avatar Apr 15 '24 07:04 moeiniamir

Hi @moeiniamir this dataset is very cool. Can you explain the MARLIN section a bit more?

bmosaicml avatar Apr 16 '24 16:04 bmosaicml

Hi @moeiniamir this dataset is very cool. Can you explain the MARLIN section a bit more?

I mistakenly created the pull request on my main branch. I just monkey-patched Marlin for internal use since it requires replacing the linear layers of the model, and I found no clean way to include that in the framework. Should I make a separate branch until the commit before that and create a new PR, or is it doable on your side?

moeiniamir avatar Apr 16 '24 16:04 moeiniamir