Prefect3: Logs created in tasks run with DaskTaskRunner dont appear in UI

Bug summary

Logs created in tasks run with DaskTaskRunner dont appear in UI

from prefect.task_runners import PrefectTaskRunner
from prefect_dask import DaskTaskRunner, get_dask_client
from dask_kubernetes.operator import KubeCluster
from prefect import flow, task
from prefect import get_run_logger

@task(log_prints=True)
def say_hello(name):
    print(f"hello {name}")
    logger = get_run_logger()
    logger.info(f"hello {name}")


@task(log_prints=True)
def say_goodbye(name):
    print(f"goodbye {name}")

@flow(task_runner=DaskTaskRunner(
    cluster_class=KubeCluster,
    cluster_kwargs={
        "name": "my-dask-cluster",
        "image": "ghcr.io/dask/dask:latest",
        "n_workers": 4,
        "env": {
            "EXTRA_PIP_PACKAGES": "prefect[dask]==3.2.15 dask_kubernetes",
            "PREFECT_API_URL": "http://...:4200/api",
            "PREFECT_LOGGING_EXTRA_LOGGERS": "dask", 
        }
    },
))
def greetings_dask(names):
    for name in names:
        say_hello.submit(name).result()
        say_goodbye.submit(name).result()

if __name__ == "__main__":
    greetings_dask(["arthur", "trillian", "ford", "marvin"])

Version info

Version:             3.2.15
API version:         0.8.4
Python version:      3.11.11
Git commit:          0f5a3081
Built:               Fri, Mar 28, 2025 3:28 PM
OS/Arch:             linux/x86_64
Profile:             ephemeral
Server type:         server
Pydantic version:    2.11.1
Integrations:
  prefect-azure:     0.4.3
  prefect-docker:    0.6.2
  prefect-shell:     0.3.1
  prefect-kubernetes: 0.5.9

Additional context

No response

hi @OliverKleinBST - do you have an MRE for this? Omitting the cluster_class specification and otherwise running your example as is, I am seeing logs in the UI against an open source server. Is it possible you're running into this and need to upgrade your server?

Hi @zzstoatzz , with omitting cluster_class I guess we have a temporary local dask server. That also works for me logging wise. The problem only happens when I connect to cluster which is created over its kubernetes operator (kubernetes-dask-operator). The same problem happens also for a ray cluster.

I am also having trouble with a dask cluster created externally (small cluster of machines on local network, no logs coming through).

With the local dask cluster I get logs, and with the local thread runner I get logs.

I tried prefect 3.3.3 and it seems to behave the same. Slightly mystified, is there a proper way to debug this? I still get errors and results from the task, so it's still communicating, just no logs.

Any update on that problem?