trulens icon indicating copy to clipboard operation
trulens copied to clipboard

[FEAT] Add python 3.12 & 3.13. support for OpenAI

Open sfc-gh-jreini opened this issue 7 months ago • 4 comments

Feature Description python 3.12 - 3.13. support

Reason Users are wanting to use TruLens on python 3.13

Importance of Feature What value does this feature unlock for you?

sfc-gh-jreini avatar May 20 '25 15:05 sfc-gh-jreini

3.12 should work already I believe. Are they running into some issue there?

sfc-gh-dkurokawa avatar May 20 '25 15:05 sfc-gh-dkurokawa

@sfc-gh-dkurokawa I reproduced this same issue on both 3.12 and 3.13: https://github.com/truera/trulens/issues/1987

sfc-gh-jreini avatar May 20 '25 16:05 sfc-gh-jreini

tagging related fix: https://github.com/truera/trulens/pull/2005

sfc-gh-jreini avatar May 27 '25 18:05 sfc-gh-jreini

I manually applied the change

diff .venv/lib/python3.10/site-packages/trulens/core/utils/pyschema.py .venv/lib/python3.10/site-packages/trulens/core/utils/pyschema.py.bak
64,66c64,66
<     Try to get the attribute `k` of the given object. If it’s a @property or
<     functools.cached_property (including OpenAI’s cached_property shim), this
<     will invoke the descriptor to compute and return the value.
---
>     Try to get the attribute `k` of the given object. This may evaluate some
>     code if the attribute is a property and may fail. In that case, an dict
>     indicating so is returned.
68,69c68,69
<     If `get_prop` is False, will raise ValueError for any property/descriptor.
<     On exceptions, returns a dict with an ERROR key.
---
>     If `get_prop` is False, will not return contents of properties (will raise
>     `ValueException`).
71c71
<     # Fetch the descriptor or raw attribute without invoking it
---
>
74,80c74,85
<     # Detect both plain @property and stdlib cached_property
<     is_prop = isinstance(v, property)
<     is_std_cached = isinstance(v, functools.cached_property)
<
<     if is_prop or is_std_cached:
<         if not get_prop:
<             raise ValueError(f"{k} is a property/descriptor")
---
>     is_prop = False
>     try:
>         # OpenAI version 1 classes may cause this isinstance test to raise an
>         # exception.
>         is_prop = isinstance(v, property)
>     except Exception as e:
>         return {constant_utils.ERROR: Obj.of_object(e)}
>
>     if is_prop:
>         if not get_prop:
>             raise ValueError(f"{k} is a property")
>
82,83c87,89
<             # Invoke the descriptor protocol to compute (and cache) the value
<             return v.__get__(obj, type(obj))
---
>             v = v.fget(obj)
>             return v
>
86,88c92,93
<
<     # Not a property/descriptor we care about—return the raw value
<     return v
---
>     else:
>         return v

The problem still exist with the fix

sqlite> select * from trulens_feedbacks;
feedback_result_hash_aa220aa3e518159076e2194b8d6564c8|record_hash_0b4b8eed912e632b73e92ca7d2841850|feedback_definition_hash_4eddfa5e9b512787b9be00e5f0048ed9|1748927718.44679|failed|Traceback (most recent call last):
  File "/Users/woody.yang/learn_llm/azure-ai-search-python-playground/.venv/lib/python3.10/site-packages/trulens/core/feedback/feedback.py", line 949, in run
    core_endpoint.Endpoint.track_all_costs_tally(
  File "/Users/woody.yang/learn_llm/azure-ai-search-python-playground/.venv/lib/python3.10/site-packages/trulens/core/feedback/endpoint.py", line 570, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
  File "/Users/woody.yang/learn_llm/azure-ai-search-python-playground/.venv/lib/python3.10/site-packages/trulens/core/feedback/endpoint.py", line 543, in track_all_costs
    return Endpoint._track_costs(
  File "/Users/woody.yang/learn_llm/azure-ai-search-python-playground/.venv/lib/python3.10/site-packages/trulens/core/feedback/endpoint.py", line 647, in _track_costs
    result: T = __func(*args, **kwargs)
  File "/Users/woody.yang/learn_llm/azure-ai-search-python-playground/.venv/lib/python3.10/site-packages/trulens/core/feedback/feedback.py", line 507, in __call__
    return self.imp(*args, **valid_kwargs)
  File "/Users/woody.yang/learn_llm/azure-ai-search-python-playground/.venv/lib/python3.10/site-packages/trulens/feedback/llm_provider.py", line 384, in context_relevance_with_cot_reasons
    return self.generate_score_and_reasons(
  File "/Users/woody.yang/learn_llm/azure-ai-search-python-playground/.venv/lib/python3.10/site-packages/trulens/feedback/llm_provider.py", line 155, in generate_score_and_reasons
    response = self.endpoint.run_in_pace(
  File "/Users/woody.yang/learn_llm/azure-ai-search-python-playground/.venv/lib/python3.10/site-packages/trulens/core/feedback/endpoint.py", line 336, in run_in_pace
    raise RuntimeError(
RuntimeError: Endpoint OpenAIEndpoint request failed 4 time(s):
	'cached_property' object has no attribute 'completions'
	'cached_property' object has no attribute 'completions'
	'cached_property' object has no attribute 'completions'
	'cached_property' object has no attribute 'completions'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/woody.yang/learn_llm/azure-ai-search-python-playground/.venv/lib/python3.10/site-packages/trulens/core/feedback/feedback.py", line 964, in run
    raise RuntimeError(
RuntimeError: Evaluation of context_relevance_with_cot_reasons failed on inputs:
{'context': 'Overview \n'
            'Introducing PerksPlus - the ultimate benefits program designed to '
            'support .
|{"calls": []}||context_relevance_with_cot_reasons|{"n_requests": 0, "n_successful_requests": 0, "n_completion_requests": 0, "n_classification_requests": 0, "n_classes": 0, "n_embedding_requests": 0, "n_embeddings": 0, "n_tokens": 0, "n_stream_chunks": 0, "n_prompt_tokens": 0, "n_completion_tokens": 0, "n_cortex_guardrails_tokens": 0, "cost": 0.0, "cost_currency": "USD"}|

yydoow avatar Jun 03 '25 05:06 yydoow