ai-on-gke icon indicating copy to clipboard operation
ai-on-gke copied to clipboard

Cannot view logs/metrics for Ray in cloud monitoring

Open AlexBulankou opened this issue 1 year ago • 0 comments

I'm calling Ray on GKE cluster via Notebook on gpt-j-online.ipynb and prediction is working as expected. I'm trying to get logging results in log explorer as per instructions, so I'm adding this to the filters:

resource.type="k8s_container"
resource.labels.cluster_name=%CLUSTER_NAME%
resource.labels.pod_name=%RAY_HEAD_POD_NAME%
resource.labels.container_name="fluentbit"

(substituting cluster_name and pod_name)

However I don't get any results at all from resource.labels.container_name="fluentbit" container. Any suggestions?

AlexBulankou avatar Aug 02 '23 22:08 AlexBulankou