Python and Cloud volume Access to FlyWire Failing

[markl02]

Hello,

I have been trying to use the fafbseg package in R (Windows 10) in order to access and analyze Flywire data, however, when trying to run functions requiring access to the data this error consistently shows up: Error in py_call_impl(callable, dots$args, dots$keywords) : AuthenticationError: No Graphene authentication token was provided. Does ~/.cloudvolume/secrets/cave-secret.json exist? You have a valid token but python+cloudvolume access to FlyWire API is still failing!

My Flywire token is stored in C:\Users\marks\Documents.cloudvolume\secrets and appears to be set correctly. Also, it appears to be related to the fact that I'm using Windows as an OS. Any help or guidance on how to resolve this would be greatly appreciated!

Thank you!

[jefferis]

Just some additional info here @william-silversmith. This is being used via the fafbseg R package via its reticulate python interface. The observed behaviour is that when accessing flywire resources directly from R, the token is successfully read from that location on disk and can be used to access private data. However from cloudvolume the token is not found. I am fairly sure this is a Windows specific issue and suspect some disagreement about how to expand the path ~/.cloudvolume/secrets/cave-secret.json or possibly a CRLF vs LF issue. Is there any way that we can get cloudvolume to report where it is trying to find the secrets file?

[jefferis]

Also @markl02 can you check the path? I think your error report on slack said

C:/Users/marks/Documents/.cloudvolume/secrets/cave-secret.json

[william-silversmith]

You can view CloudVolume's secret path as follows:

import cloudvolume.secrets
path = cloudvolume.secrets.secretpath('secrets/cave-secret.json')

That might reveal the problem already... I should be using os.path.join instead of using a '/'.

[william-silversmith]

Yes, in my windows 10 emulator, I get: C:\Users\username\.cloudvolume\secrets/cave-secret.json That may well be the source of the problem.

[jefferis]

Thanks, Will! That mixed path does look unhappy.

@markl02 can you check this on your machine what the R+python combo reports by doing this?

secrets=reticulate::import('cloudvolume.secrets')
secrets$secretpath('secrets/cave-secret.json')

[markl02]

That outputs:

"C:\Users\marks\.cloudvolume\secrets/cave-secret.json"

[william-silversmith]

I have a fix for this issue pushed to master. I'm waiting for some automated tests to run, but then I'll try pushing out a bugfix release.

[jefferis]

Ah, thanks a lot, Will. But there may be an additional problem here besides that forward/reverse slash.

If we ask R and python/cloudvolume to expand ~/.cloudvolume/secrets/cave-secret.json on Mark's machine:

# R says
C:/Users/marks/Documents/.cloudvolume/secrets/cave-secret.json

# cloudvolume/python says
C:\Users\marks\.cloudvolume\secrets/cave-secret.json

Any suggestions re the difference?

[jefferis]

@markl02 can you report what happens if you do

# Python via reticulate
os=reticulate::import('os')
os$path$expanduser("~/.cloudvolume/secrets/cave-secret.json")

# native R
path.expand("~/.cloudvolume/secrets/cave-secret.json")

[william-silversmith]

It seems the windows equivalent of $HOME may be configured differently in those situations. One possible solution is to place the cave-secret (possibly as a symbolic link) in both locations as there is probably a reason for R having a different home.

[william-silversmith]

You can also explicitly set the environment variable setx CLOUD_VOLUME_DIR "C:\Users\marks\.cloudvolume" which should override that logic.

[markl02]

Here's the output of that @jefferis :

> secrets=reticulate::import('cloudvolume.secrets')
> secrets$secretpath('secrets/cave-secret.json')
[1] "C:\\Users\\marks\\.cloudvolume\\secrets/cave-secret.json"
> # Python via reticulate
> os=reticulate::import('os')
> os$path$expanduser("~/.cloudvolume/secrets/cave-secret.json")
[1] "C:\\Users\\marks/.cloudvolume/secrets/cave-secret.json"
> 
> # native R
> path.expand("~/.cloudvolume/secrets/cave-secret.json")
[1] "C:/Users/marks/Documents/.cloudvolume/secrets/cave-secret.json"

[jefferis]

Thanks. So Mark if you set CLOUD_VOLUME_DIR environment variable on the R side to

Sys.setenv(CLOUD_VOLUME_DIR=path.expand("~/.cloudvolume"))

then maybe you could force them to agree?

Will, when does cloudvolume check this environment variable. Once on initialisation?

[william-silversmith]

Yes that's right. The soon as import cloudvolume is run, the variable is initialized.

[jefferis]

OK @markl02 can you start a clean R session, do

Sys.setenv(CLOUD_VOLUME_DIR=path.expand("~/.cloudvolume"))
library(fafbseg)
dr_fafbseg()

and report back?

[markl02]

After doing that I still get the same exact error message:

> dr_fafbseg()
FlyWire
----
Valid FlyWire ChunkedGraph token is set!
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  AuthenticationError: No Graphene authentication token was provided. Does ~/.cloudvolume/secrets/cave-secret.json exist?
You have a valid token but python+cloudvolume access to FlyWire API is still failing!
Please ask for help at https://groups.google.com/g/nat-user using the full output of dr_fafbseg.

1 FlyWire/CloudVolume credential files available at
C:/Users/marks/Documents/.cloudvolume/secrets
[1] "cave-secret.json"

[jefferis]

Mark, can you report

cv=fafbseg:::check_cloudvolume_reticulate()
cv$secrets$CLOUD_VOLUME_DIR
fs::path_expand("~/.cloudvolume")

For reference this seems to be an unusual decision of R on Windows about how to define the home dir. Discussed in https://rdocumentation.org/packages/fs/versions/1.5.2/topics/path_expand.

[markl02]

> cv=fafbseg:::check_cloudvolume_reticulate()
> cv$secrets$CLOUD_VOLUME_DIR
[1] "C:\\Users\\marks\\.cloudvolume"
> fs::path_expand("~/.cloudvolume")
C:/Users/marks/.cloudvolume

[jefferis]

OK, so I don't understand why but it seemed that CloudVolume did not pick up the CLOUD_VOLUME_DIR environment variable for some reason. Nevertheless I think that I should try to get R on Windows to behave more like Python on Windows and make a change over in fafbseg.

[markl02]

Thank you both so much for all your assistance! I'm wondering if any progress has been made in fixing this issue?

[jefferis]

Hi @markl02, as a rather crude workaround for now you can make a symbolic link or just copy your cave-secret.json file so that it is visible at both

C:/Users/marks/Documents/.cloudvolume/secrets

and

C:/Users/marks/.cloudvolume/secrets

A better fix will have to await my return from annual leave.