airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Fetch intermediate log async GKEStartPod

Open pankajastro opened this issue 9 months ago • 3 comments

This PR introduces a parameter that enables the retrieval of intermediate logs for the GKEStartPod asynchronous operator.

  • Add param last_log_time and logging_interval in GKEStartPodTrigger serialize
  • Add optional param last_log_time in method invoke_defer_method

Example DAG:

start_pod = GKEStartPodOperator(
        task_id="start_pod",
        project_id=PROJECT_ID,
        location=LOCATION,
        cluster_name=GKE_CLUSTER_NAME,
        do_xcom_push=True,
        namespace=GKE_NAMESPACE,
        image="ubuntu:jammy",
        cmds=["sh", "-c", "timeout 300 bash -c 'while true; do echo \"meow\"; sleep 30; done'"],
        name="test-sleep",
        in_cluster=False,
        on_finish_action="delete_pod",
        deferrable=True,
        get_logs=True,
        logging_interval=5,
        gcp_conn_id=GCP_CONN_ID
    )
Screenshot 2024-05-01 at 6 53 50 PM

^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

pankajastro avatar May 01 '24 13:05 pankajastro

Any updates on this?

jonathonbattista avatar May 07 '24 21:05 jonathonbattista

Any updates on this?

I'll try address the pending review by end of this week

pankajastro avatar May 08 '24 08:05 pankajastro

Bump on this 🙏

jonathonbattista avatar May 14 '24 19:05 jonathonbattista

@pankajastro Any ETA on this? Thanks

jonathonbattista avatar May 24 '24 13:05 jonathonbattista

@pankajastro Any ETA on this? Thanks

CI is green now. Hopefully, we will merge soon. In the meantime, could you please test it once?

pankajastro avatar May 25 '24 07:05 pankajastro