[Roadmap] Logging and Observability

[sonichi]

We want the internal events in agents and conversation patterns to be logged. We also want to support integration with external loggers.  

### Tasks
- [x] Track each llm request and response [PR #1146]
- [ ] Add telemetry ID to agent chat message
- [ ] Logging for streaming mode
- [ ] Logging for AgentEval
- [ ] Add verbose level for the logging
- [ ] Extra logging for teachable agent
- [ ] Extra logging for group chat
- [ ] Test logging with local models
- [ ] https://github.com/microsoft/autogen/issues/770
- [ ] Profile cost and latency of test executio
- [ ] https://github.com/microsoft/autogen/issues/2423
- [ ] https://github.com/microsoft/autogen/pull/2478
- [ ] https://github.com/microsoft/autogen/issues/2581
- [ ] https://github.com/microsoft/autogen/issues/2670

[sonichi]

After solving #203 logging can be done in a new way.

[victordibia]

This is related to #412 where we are discussing the structure of the content of the logged_history. Given the overlap, I'll work on that when #203 is finalized.

[radman-x]

#203 appears to be done. When will we have logging in agents restored? This is VERY important.

[sonichi]

@victordibia it'll be nice to share progress if you know any or ask the dev working on this issue to share progress.

[victordibia]

@afourney , is there a branch we can link here on what Cheng is doing? I agree with @sonichi that sharing progress will provide surface to provide feedback and also avoid duplicate work.

[afourney]

Not yet that I am aware of. Let's not develop anything here until we hear from her tomorrow

[cheng-tan]

Hi folks, I'm adding in the new logging for each agent, will have more updates this week.

[ekzhu]

@cheng-tan maybe create a draft PR so we are aware of this work item?

[cheng-tan]

Hi @radman-x, logging changes have been released with autogen v0.2.14 yesterday, documentation and example notebook. Please let me know if you have any questions or feedback, thanks.

[lalo]

I've been exploring with this notebook to consume the sqlite trace - sort of relevant to this issue: https://github.com/microsoft/autogen/pull/2526

[Hk669]

i've made a PR on custom logger functionality, and the FileLogger #2596

[robinmanuelthiel]

Is it the plan to support Open Telemetry, especially for tracing?