[Bug]: "response_cost" in kwargs of async_log_success_event is at sometimes randomly 0

[jeromeroussin]

What happened?

My cicd pipeline is seeing numerous failures with 1.43.15 when checking that a cost is properly passed in to my custom success callback. The same test, using the same model will sometimes see a proper positive cost value, and sometimes just 0. My test calls a gpt-4 chat model.

My callback does not do much but this with the response_cost:

class MyCustomHandler(CustomLogger):
    ...
    async def async_log_success_event(self,kwargs,response_obj,start_time, end_time):
     ...
      payload = {
                  'model'                  : kwargs['model'],
                  ...
                  'cost'                   : kwargs['response_cost'],
    ...

This behavior is not consistent. Sometimes a rerun of the pipeline will not see it and pass.

Relevant log output

2024-08-16 11:17:22 {'model': 'gpt-4-0314', 'model_mode': 'chat', 'location': 'Azure East US', 'key_name': 'test_application', 'key_owner': 'test', 'user': 'renovatebot', 'start_time': '2024-08-16 11:16:55.029494', 'end_time': '2024-08-16 11:17:17.083287', 'duration': 22.053793, 'cost': 0, 'prompt_tokens': 18, 'total_tokens': 324, 'client_address': None, 'namespace': 'cicd', 'completion_tokens': 306, 'messages': [{'role': 'user', 'content': 'Write a function that prints n primes in python and javascript'}], 'response': "Python:\n\n\ndef print_n_primes(n):\n    count = 0\n    num = 2\n    while count < n:\n        for i in range(2, num):\n            if (num%i) == 0:\n                break\n        else:\n            print(num)\n            count += 1\n        num += 1\n\nprint_n_primes(10)\n\n\nThe above function prints the first `n` prime numbers in Python. The `for` loop checks each number for primality, and if it is prime, it prints the number and increments the count.\n\nJavascript:\n\n```javascript\nfunction print_n_primes(n) {\n    var count = 0;\n    var num = 2;\n    while(count < n) {\n        var isPrime = true;\n        for(var i = 2; i < num; i++) {\n            if(num % i == 0) {\n                isPrime = false;\n                break;\n            }\n        }\n        if(isPrime) {\n            console.log(num);\n            count++;\n        }\n        num++;\n    }\n}\n\nprint_n_primes(10);\n```\n  \nThe function in JavaScript works similarly. The only difference is instead of python's `else` condition outside `for` loop, we are using the `isPrime` boolean variable. If no number divides the current number (`num`), then `isPrime` stays `true`, and the number is logged to the console, and the `count` is incremented."}

Twitter / LinkedIn details

https://www.linkedin.com/in/jeromeroussin/