Compute Engine Metadata server unavailable when VM instance preempties

[Trollgeir]

I am using a shutdown script to save a file whenever preemptible VM instances shut down.

I am authed to my project inside the VM through service accounts (cloud), as I have access by simply doing fs = gcsfs.GCSFileSystem() during runtime.

However, at the time when the VM preempties and my shutdown script triggers, I cannot instantiate the filesystem anymore: Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out..

Does this mean I have lost all internet connectivity, or just some? Would it work to somehow cache my authentication?

[martindurant]

Sorry, I don't know the chain of operations when a VM is shut down. It would make sense to have processes notified before metadata services disappear, but you would need to ask google. It is possible that you can create the gcsfs instance before the shutdown process and have its token still be valid, but I haven't tried this either - maybe all creds are invalidated when shutdown begins.

[Trollgeir]

Thanks for your response. Do you know if the token is cached, or if it gcsfs tries to acquire a new token every time I instantiate a filesystem?

[martindurant]

If an instance has been created, it keeps hold of it's credentials; and that instance will get reused by any other code using the same connection arguments. Presumably that token can expire.