Lance comments

Results 8 comments of


Lance

How can I train it on multi-GPU

@jrabary is there any update on the multi-gpu inference + training strategies ? do you have any example of getting it to work correctly? I'm running into a few issues...

'GrpcServerResourceHandleOp' is neither a type of a primitive operation nor a name of a function registered in binary running on n-b0fdb3cc-w-0.

Hi, Thank you for your detailed answer. Do you think I could get a hold of that private TPU custom op or custom gcp tpu vm ? I assume it...

'GrpcServerResourceHandleOp' is neither a type of a primitive operation nor a name of a function registered in binary running on n-b0fdb3cc-w-0.

it has been tried by our team, unfortunately, the grpc binaries cannot be loaded with the provider version of TensorFlow nightly (it is missing required libtensorflow framework bins), and other...

'GrpcServerResourceHandleOp' is neither a type of a primitive operation nor a name of a function registered in binary running on n-b0fdb3cc-w-0.

we already have recompiled the GRPC library, the TPU VMS work with a modified version of `tf-nightly`. simply pulling or building tf from source will not cut it even for...

'GrpcServerResourceHandleOp' is neither a type of a primitive operation nor a name of a function registered in binary running on n-b0fdb3cc-w-0.

small update, I am still getting errors throughout my testing trace this is with a custom built tensorflow from source (and grpc.so accordingly) using the flags given by the working...

'GrpcServerResourceHandleOp' is neither a type of a primitive operation nor a name of a function registered in binary running on n-b0fdb3cc-w-0.

The logs were from the TensorFlow built from the source, the preinstalled TF (wheel located at /usr/share/tpu/) works great, but doesn't support custom ops. I guess it's just the master...

'GrpcServerResourceHandleOp' is neither a type of a primitive operation nor a name of a function registered in binary running on n-b0fdb3cc-w-0.

still no luck as of 18/06, any help would be appreciated. I have tried recompilation with tpu code build modifications but I wasn't able to reproduce what makes the given...

'GrpcServerResourceHandleOp' is neither a type of a primitive operation nor a name of a function registered in binary running on n-b0fdb3cc-w-0.

hey @michaelbanfield Thanks for the update, I was away for a vacation. Couldn't answer. Unfortunately, it looks like it has fixed the GRPC problem, but didn't fix the entirety of...