[Feature Request] Hope to get the token usage report per AOAI connection deployment

[Wix97]

For now the token usage report is summarized per batch run. However, it would be more helpful for users to estimate costs if promptflow could provide token usage reports for each AOAI connection deployment.

It would also be helpful if the user can read such report in json format via PF SDK.

[brynn-code]

Hi @Wix97 . Thanks for reaching us, could you please elaborate more about the 'usage of each AOAI connection deployment'? Is that related to any run, for example a batch run with different deployment, to get the usage of each deployment, or just overall token usage of a deployment? For the latter one, seems that should be the capability of AzureOpenAI instead of promptflow.

[Wix97]

Sorry for the confusion, the first case is what I wanted to mention here. For example, there is a flow consuming 2 AOAI connections, each connection has 2 deployments. It would be helpful if promptflow can generate token usage report such like:

  "token_usage": {
    "connection_1": {
      "connection_1_deployment_1": {
        "prompt_tokens": 123,
        "completion_tokens": 456
      },
      "connection_1_deployment_2": {
        "prompt_tokens": 123,
        "completion_tokens": 456
      }
    },
    "connection_2": {
      "connection_2_deployment_1": {
        "prompt_tokens": 123,
        "completion_tokens": 456
      },
      "connection_2_deployment_2": {
        "prompt_tokens": 123,
        "completion_tokens": 456
      }
    }
  }
}

[thy09]

@Wix97 From pf perspective, it would be complicated to define such a contract to summarize tokens according to the tokens.

I have a proposal: what if we expose the ability to get the tokens in other nodes? For example, we provide such an experession: {llm_node_1.system_metrics.usage} to get the usage dictionary of a node, then you can have a summary node to get the dictionary for yourself.

[Wix97]

LGTM👍