paxml icon indicating copy to clipboard operation
paxml copied to clipboard

Published 20 hours ago • verified publisher icon google

DEADLINE_EXCEEDED on 1024 GPUs.

Open mhugues opened this issue 2 months ago • 0 comments

Additional GRPC error information from remote target unknown_target_for_coordination_leader while calling /tensorflow.CoordinationService/RegisterTask: :{"created":"@1712965181.656280441","description":"Deadline Exceeded","file":"external/com_github_grpc_grpc/src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4} 2024-04-12 23:39:41.656900: E external/xla/xla/pjrt/distributed/client.cc:96] Coordination service agent in error status: DEADLINE_EXCEEDED: Deadline Exceeded

Did anyone see that issue?

mhugues avatar Apr 16 '24 20:04 mhugues