cylc
Make API info available - CLI via Hub/proxy and/or UIS
These changes partially address https://github.com/cylc/cylc-flow/issues/5235 Sibling to https://github.com/cylc/cylc-flow/pull/5267
On spawning the UI Server (Jupyter server), the hub negotiates an API token and puts it as JUPYTERHUB_API_TOKEN, along with other variables, into the the environment. If not hub spawned the UI Server will generate one itself.
Using the server app's information (self.serverapp.server_info()) and overwriting it with HUB info, we can make this available by writing it to a (user read-only) file:
$ pwd
/home/sutherlander/.cylc/uiserver
$ ls -larth api_info.json
-rw------- 1 sutherlander sutherlander 288 Nov 22 11:25 api_info.json
$ cat api_info.json
{"url": "http://127.0.0.1:39409/user/sutherlander/", "hostname": "127.0.0.1", "port": 39409, "sock": "", "secure": false, "base_url": "/user/sutherlander/", "token": "2f2abf2e4bcd43589711c1554ce2cc7e", "root_dir": "/home/sutherlander", "password": false, "pid": 6610, "version": "1.21.0"}
(this token changes on each UIS spawn)
This can be used to access the UIS via the UIS url (as above) but also via the hub proxy (http://localhost:8000/user/sutherlander/)..
And copied (manually only?) to machines on the network for API access there.
Tasks:
- [x] - Write API info, including hub token if available, to a user read-only file.
- [ ] - Clean up file on UI Server shutdown (use absence as indicator that UIS isn't running?).
-
- [x] Hubless clean-up.
-
- [ ] Hub stopped server clean-up (https://github.com/cylc/cylc-uiserver/issues/392)
- [ ] - Find out hub proxy URL (if available), and use that instead or in addition to UIS URL.
Sample:
#!/usr/bin/env python3
import json
import requests
from cylc.uiserver.app import API_INFO_FILE
f = open(API_INFO_FILE, "r")
api_info = json.loads(f.read())
f.close()
query = '''
query {
workflows {
id
stateTotals
}
}
'''
r = requests.post(api_info["url"] + 'cylc/graphql',
headers={
'Authorization': f'token {api_info["token"]}',
},
json={'query': query}
)
r.raise_for_status()
data = r.json()
print(json.dumps(data, indent=4))
$ ./uis_api.py
{
"data": {
"workflows": [
{
"id": "~sutherlander/linear/run1",
"stateTotals": {
"waiting": 2,
"expired": 0,
"preparing": 0,
"submit-failed": 1,
"submitted": 0,
"running": 0,
"failed": 0,
"succeeded": 0
}
}
]
}
}
Requirements check-list
-
[x] I have read
CONTRIBUTING.mdand added my name as a Code Contributor. -
[x] Contains logically grouped changes (else tidy your branch by rebase).
-
[x] Does not contain off-topic changes (use other PRs for other changes).
-
[ ] Appropriate tests are included (test for token use should be included).
-
[ ] Already covered by existing tests (file is written on UIS start).
-
[ ] Appropriate change log entry included.
-
[ ] I have opened a documentation PR at cylc/cylc-doc/pull/XXXX.
Codecov Report
Base: 80.00% // Head: 79.84% // Decreases project coverage by -0.15% :warning:
Coverage data is based on head (
6765e1e) compared to base (c204b88). Patch coverage: 80.95% of modified lines in pull request are covered.
Additional details and impacted files
@@ Coverage Diff @@
## master #396 +/- ##
==========================================
- Coverage 80.00% 79.84% -0.16%
==========================================
Files 12 12
Lines 1160 1181 +21
Branches 197 199 +2
==========================================
+ Hits 928 943 +15
- Misses 195 198 +3
- Partials 37 40 +3
| Impacted Files | Coverage Δ | |
|---|---|---|
| cylc/uiserver/app.py | 87.40% <80.95%> (-1.19%) |
:arrow_down: |
| cylc/uiserver/authorise.py | 88.64% <0.00%> (-0.88%) |
:arrow_down: |
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.
:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.
Thanks for looking at this @datamel !
Oh that's handy to know, found the hub created one.. a whole pile of them (~600 or so)
We should use it, it worked setting os.environ["JUPYTER_RUNTIME_DIR"] in the hubapp.py ..
$ ls .cylc/uiserver/info_files/
jpserver-7002.json jpserver-7002-open.html jupyter_cookie_secret
However, a couple of things:
- How do we set it so it uses a relative path (i.e.
~/.cylc...), or evaluates the users home directory each time? - How do we get it to clean up the files when stopped by the hub? (at the moment they just collect in that
.local/share/jupyterand are never cleaned up)
We could possibly just do something like:
list_of_paths = folder_path.glob('*.json')
latest_path = max(list_of_paths, key=lambda p: p.stat().st_ctime)
(found on some substack) to use the latest UIS.
Another thing is, I want to be able to use the hub proxy URL instead of UIS directly (as that may be exposed to a different network).. But I need to be able to scrape the URL from somewhere, and probably add it to this file.. Any ideas?
Oh that's handy to know, found the hub created one.. a whole pile of them (~600 or so)
Yes, Jupyter do clean them but I think we need to add our own clean logic as it seems patchy (this was discussed recently on element). They don't get cleaned when cylc gui/hub is not shut down cleanly, e.g. killed with a kill -9.
I think when we reuse these files, it may be necessary to check that the process is still running and then clean the files if it isn't and reselect until we get a working one. That should be a quick pr to do as we have the details of the process number in the file name. Although I think some investigation needs to happen in terms of what host the process is on and where the file is - is a file also generated for each user connecting to the hub?
We could possibly just do something like:
list_of_paths = folder_path.glob('*.json') latest_path = max(list_of_paths, key=lambda p: p.stat().st_ctime) (found on some substack) to use the latest UIS.
At the moment we do: https://github.com/cylc/cylc-uiserver/blob/c42857b233888f7bea8e63475bccf22837de54cb/cylc/uiserver/scripts/gui.py#L58 and then randomly select one, rather than using the latest file (note that this file is the html open one, not the json one that you need - they are generated in a pair). I don't fully understand the issue of implementing this comms method - would any uiserver info be suitable, or do we need the particular uiserver that the workflow will be going through?
We should use it, it worked setting
os.environ["JUPYTER_RUNTIME_DIR"]in thehubapp.py..
Just thinking now that we may want to put them in a separate dir to the one used for cylc gui files? Since we are reusing guis (and updating the html file with a new url) now I think it best to keep these separate. This will stop us selecting a hub file when we want to open the gui (specifically not going through the hub). Perhaps we could redesign the structure of the dir so we keep gui and hub files in their own sub dirs? ~/.cylc/uiserver/info_files/hub/ and ../gui/?
Another thing is, I want to be able to use the hub proxy URL instead of UIS directly (as that may be exposed to a different network).. But I need to be able to scrape the URL from somewhere, and probably add it to this file.. Any ideas?
I'll read up about this and get back to you. I'm not fully sure how to do this at the moment!
Just thinking now that we may want to put them in a separate dir to the one used for cylc gui files? Since we are reusing guis (and updating the html file with a new url) now I think it best to keep these separate.
The hub located files and the other files look to have identical infomation (not hub specific, tho might pay to check), so I think it might be safe to locate these in the same place.
However, for the CLI (and maybe the gui), I would like to have the hub proxy URL available to use (maybe preferentially).. And it would help if we didn't have to read two files (i.e. have the same file contain hub_url field)..
I would go further to say, instead of starting up a new UIS for the gui, we should reuse existing for gui, no? (regardless of hub started or non-hub)
I don't fully understand the issue of implementing this comms method - would any uiserver info be suitable, or do we need the particular uiserver that the workflow will be going through?
Any UIS works for the CLI via UIS, and we can also use the hub proxy url (which may be preferable, as it might be intentionally exposed to a wider network)
They don't get cleaned when cylc gui/hub is not shut down cleanly, e.g. killed with a kill -9.
Also, they are not cleaned up when stopping the UIS via the hub interface...