Agent Log Capture Feature

[Future-Outlier]

Tracking issue

https://github.com/flyteorg/flyte/issues/3936

Why are the changes needed?

Make every log in the agent server can be shown in flyte console instead of the agent server.

What changes were proposed in this pull request?

In airflow agent, use io.StringIO and "logging" module to capture logging messages, and send them to flytepropeller.

TODO: we should split the feature into 2 functions, set log capture and unset it. we can move it to flytekit.extend.backend.utils, to support all agents.

How was this patch tested?

In local single binary.

if message:
    message = message + "\n" + log_capture_string.getvalue()
else:
    message = log_capture_string.getvalue()
print("message: ", message)
return GetTaskResponse(resource=Resource(phase=cur_phase, message=message))

Setup process

Screenshots

Check all the applicable boxes

• [ ] I updated the documentation accordingly.
• [x] All new and existing tests passed.
• [x] All commits are signed-off.

Related PRs

Docs link

[Future-Outlier]

I've faced a problem on how to close the airflow logger on the terminal, not yet found a solution. Also, I am finding ways to show message on flyteconsole in RUNNING Phase, need to tweak some code in flytepropeller.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 83.08%. Comparing base (95ea92d) to head (c4fce7d).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2213      +/-   ##
==========================================
- Coverage   85.83%   83.08%   -2.75%     
==========================================
  Files         314      177     -137     
  Lines       24027    17549    -6478     
  Branches     3641     3641              
==========================================
- Hits        20623    14581    -6042     
+ Misses       2762     2326     -436     
  Partials      642      642              

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.