trulens icon indicating copy to clipboard operation
trulens copied to clipboard

Add Vectara Hallucination Detection Model

Open Josephrp opened this issue 1 year ago • 6 comments

Added vectara hallucination detection model to the huggingface class

adding an exposition/model example using huggingface models end to end for demo

this is a draft PR , just need to add some text to explain the examples we have chosen


Ellipsis :rocket: This PR description was created by Ellipsis for commit 4f3353e23a27528c1075b182753d15da0667ce32.

Summary:

The PR adds a new method hallucination_evaluator to the Huggingface class in hugs.py for evaluating the hallucination score of a combined input of two statements using the Huggingface hallucination evaluation model.

Key points:

  • Added hallucination_evaluator method to Huggingface class in hugs.py.
  • The method uses the HUGS_HALLUCINATION_API_URL endpoint for the hallucination evaluation model from Huggingface.
  • The method takes two arguments: model_output and retrieved_text_chunks, combines them, and sends a POST request to the API.
  • The response is parsed to extract the hallucination score.

Generated with :heart: by ellipsis.dev

Josephrp avatar Mar 02 '24 11:03 Josephrp

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

Can you remove the .ipynb checkpoints from this change?

Also can you elaborate on why there are two separate notebooks to show this capability?

joshreini1 avatar Mar 04 '24 20:03 joshreini1

Thanks for all the work on this @MN-Noor @Josephrp - this is close! Just added a few small comments before it can go in

joshreini1 avatar Mar 04 '24 20:03 joshreini1

hi @MN-Noor would you like to take the last reviews above ?

@joshreini1 , thanks for the comments, should we keep both notebooks renamed or remove one or the other?

sorry to bother you but i'll be able to wrap this up :-)

Josephrp avatar Mar 09 '24 12:03 Josephrp

@Josephrp keep the one in /models and remove the other. Thanks!

joshreini1 avatar Mar 09 '24 13:03 joshreini1

@joshreini1 thanks ! @MN-Noor wrapped it up nicely , hope that's us done - until next time !

Josephrp avatar Mar 13 '24 17:03 Josephrp

@Josephrp @MN-Noor do you have a Twitter/x account? I’ll shoutout from the TruLens handle if so!

joshreini1 avatar Mar 23 '24 16:03 joshreini1

ha! i sure do , here's me . Noor's doesnt work well where she is , so that's why she prefers linked-in

hope that's okay :-)

Josephrp avatar Mar 23 '24 16:03 Josephrp