Surfacing intermediate events (e.g. tool calls, LLM generations, etc.) in a long-running flow.

[saqadri]

Currently, we have a logger that emits log events, but there's no way to subscribe to specific events to show progress to a user while an llm.generate(..) call is ongoing. For e.g. this https://github.com/lastmile-ai/mcp-agent/blob/main/examples/mcp_basic_agent/main.py#L37 will result in multiple tool calls but only the final result will be available to the caller.

One approach (suggested by @sherryyshi) is having scopes for every event, and then allow a caller to filter by specific scopes (we already have a mechanism to subscribe to events here: https://github.com/lastmile-ai/mcp-agent/blob/main/src/mcp_agent/logging/listeners.py#L38)

Some early o1 thoughts on this: https://chatgpt.com/share/6796a263-b6e4-8011-aa4b-473fc423aaa9

[saqadri]

@evalstate fyi -- so we can move the discussions on this to public GH issues

[evalstate]

signposting https://github.com/evalstate/mcp-agent/tree/feat/progress-reporting for anyone interested in this

[evalstate]

https://github.com/lastmile-ai/mcp-agent/pull/18

[mattmorgis]

Hi! I see #18 was merged, but this is still open.

I was exploring some changes to the logging and figured I would check here before submitting a PR.

First off was the ability to log to both console and a file.

And second was to make a new file. This could be specific to my use case, but I am using a workflow pattern so a log file per each workflow run would be helpful than letting the current one grow infinitely.

Any thoughts/feedback?

[saqadri]

Hi! I see #18 was merged, but this is still open.

I was exploring some changes to the logging and figured I would check here before submitting a PR.

First off was the ability to log to both console and a file.

And second was to make a new file. This could be specific to my use case, but I am using a workflow pattern so a log file per each workflow run would be helpful than letting the current one grow infinitely.

Any thoughts/feedback?

@MattMorgis I'll let @evalstate offer suggestions on structuring logging and will leave this issue open for the time being. I would be open to further contributions on improved logging functionality. I just published v0.0.8, which includes changes from #18, and there's an additional PR coming that I am currently reviewing (#29), but neither covers your desire to have logging/reporting be split across multiple files etc.

[evalstate]

Hi @MattMorgis -- couple of quick questions...

1. When you say both console and file, I assume that means simultaneously?
2. Do you envision the file name being autogenerated per-run (e.g. YYYYMMDDHHMMSS) or specified via a switch of some sort?

To be honest, I normally run with type=file and level=error don't often see them :)

[mattmorgis]

1. When you say both console and file, I assume that means simultaneously?
2. Do you envision the file name being autogenerated per-run (e.g. YYYYMMDDHHMMSS) or specified via a switch of some sort?

Yes to both! I have this working locally:

  transports: [console, file]
  level: debug
  
  path_pattern: "logs/my-mcp-app-{timestamp}.jsonl"
  timestamp_format: "%Y%m%d_%H%M%S" 

[evalstate]

that sounds cool - is it/could it be backwards compatible? for example

transport: file
transport: [file,console] 

and keep existing path behaviour with path_pattern as an override?

[mattmorgis]

Yep! I had just kept path.

I just submitted #42 if you wanna take a look