FActScore
FActScore copied to clipboard
shmsw25
A package to evaluate factuality of long-form generation. Original implementation of our EMNLP 2023 paper "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation"
I have created a Python package for extracting facts from texts and scoring them based on knowledge sources. This package is maintained and up-to-date with the latest OpenAI APIs. You...
Hi, i have inserted some updates into the code to improve speed and also make it compatible with the latest openai API updates. Could you have a look and see...
Hi, i would like to ask if it has been tried out with llama that batch inference works? i followed this https://huggingface.co/docs/transformers/llm_tutorial#wrong-padding-side , where they pass both the input_ids and...
Hi, I have access to Llama through Huggingface and am trying to run FactScore through the CLI: `python -m factscore.factscorer --input_path data/unlabeled/InstructGPT.jsonl --model_name retrieval+llama+npm` Shouldn't this use llama to do...
after download the data and set the environment, I run this command python -m factscore.factscorer --input_path "/root/FNDLLM/test.jsonl" --model_name "retrieval+llama+npm" --use_atomic_facts --data_dir '/root/.cache/factscore/ and get this File "/root/anaconda3/envs/factstore/lib/python3.7/site-packages/factscore/retrieval.py", line 57, in...
@martiansideofthemoon here's the code i used to make atomic fact generation use the chat api rather than the legacy completion api
@martiansideofthemoon here are the dependency changes i mentioned to the pyproject.toml. It might be possible to further loosen the requirements, tough supporting `torch>2.0` and getting rid of the mandatory install...
AssertionError: `topic` in your data (Jessie Mae Brown Beavers) is likely to be not a valid title in the DB.
Good morning, I was trying to use the factscorer function. With this code : I received this message error : If I remove the `openai_key=None` in `factscore = FactScorer(model_name="retrieval+llama", data_dir="/kaggle/working/")`,...
Now that OpenAI has deprecated the Completion API, we should move our code to ChatCompletion: https://github.com/shmsw25/FActScore/blob/main/factscore/openai_lm.py#L75
