Closes #5091 #3078 #817

I have:

[ ] Formatted any Python files with black
[ ] Brought the branch up to date with master
[ ] Added any relevant Github labels
[ ] Added tests for any new additions
[ ] Added or updated any relevant documentation
[ ] Added an Architectural Decision Record (ADR), if appropriate
[ ] Added an MPLv2 License Header if appropriate
[ ] Updated the Changelog

Description

This PR reworks the way FlowAPI and FlowAuth handle query-scoped permissions. At present, the permissions scopes are generated from a full walk of every set of query trees at every geographic level. This was causing memory issues with the growing number of queries, so this pull request implements a new system.

In this PR, FlowAuth consists of four entities; Servers, Users, Roles and Scopes.

erDiagram
    Servers ||--|{ Roles : contains
    Roles }|--|{ Scopes : "provides permissions for"
    Users }|--|{ Roles : "can work with"
    Servers ||--|{ Scopes : "provides"

Scopes are applied at the user level, giving permission to:

read results
run any queries at all
use specified queries, aggregations and location metrics to build their queries
(potentially) define the date range used to run their queries. Scopes are automatically populated from a Server when it is registered or updated via a new FlowAPI endpoint Roles are set by the FlowAuth admin, and consist of a set of Scopes. Roles are specific to Servers. Users are assigned Roles by the FlowAuth admins.

The new auth flow runs as below:

sequenceDiagram
    participant Flowauth
    actor User
    participant FlowAPI
    note over User: Selects roles for token
    User ->>+ Flowauth: roles
    note over Flowauth: Checks user is permitted roles
    Flowauth ->>+ User: signed token w/ roles + scopes
    note over User: Builds query
    User ->>+ FlowAPI: query, token
    note over FlowAPI: Checks query params are covered by a single role
    FlowAPI ->>+ User: query results

This PR will remove Groups and Capabilities from FlowAuth, being replaced with Roles and Scopes. Tokens will also be removed as a database entitiy; instead, FlowAuth will treat a Token as a view of a combination of a User, a set of Roles and an expiry date.

May 20 '22 15:05 Thingus

Test summary

4 • 0 • 0 • 0 • 0

Run details


Project	FlowAuth
Status	Passed
Commit	cf93b8d11c
Started	Dec 22, 2022 12:55 PM
Ended	Dec 22, 2022 12:56 PM
Duration	01:09 💡
OS	Linux Debian -
Browser	Electron 106

View run in Cypress Dashboard ➡️

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

May 20 '22 16:05 cypress[bot]

UserObject.can_get_available_dates() will need to be updated to check for the new 'get_available_dates' scope, rather than the old 'get_result:available_dates'.

-In branch now -John

Aug 12 '22 16:08 jc-harrison

Permissions system pull request + changelist

(created by referring to git diff master...permissions_rework -- 'flowauth/backend' ...subbing in for folder path as appropriate)j

Overarching changes

Permissions are now granted on 'role' - a 'role' is a collection of 'scopes'
- A role has an expiry date after which it cannot be used to grant scopes
- A role has a maximum duration that tokens granted using it can last
Each role is tied to one server.
A 'user' has a set of 'roles'
'Groups' have been removed
'scopes' are now either simple or complex (still thinking on that)
Simple scopes are one of the following strings: ('run', 'get_available_dates', 'get_results') and represent a server-wide capacity granted by that role.
Complex scopes are a string: '[agg_unit]:[top-level scope]:[inner scope]' granting access to a particular Flowmachine query
- 'agg_unit'; An aggregation unit ('admin1 .. admin3, latlon, 'nonspatial') ('nonspatial' wwill be implemented on merge #5278#)
- 'top_level query' is a flowmachine ExposedQuery
- 'inner query' is a child query of ExposedQuery. Each ExposedQuery contains itself as an inner query, (ex. 'admin1:daily_users:daily_users')

FlowAPI changes

'scopes' now nested inside 'roles' in tokens

/flowapi/flowapi/api_spec.py

Default scopes are added to role here (53-55)

flowapi/config.py

Default JWT_IDENTITY_CLAIM changed to 'sub' in line with JWT recs

flowkit/jwt_auth_callbacks.py

identitiy key replaced with 'sub'

flowkit/permissions.py

schema_to_scopes now works with the new scope system
Rest has been rewritten to accomodate the new scope format

flowkit/user_model.py

UserObject now takes a scopes dictionary
has_access now works with the new roles, checking that one role encompasses all requested scopes in the token

Flowauth backend changes

Roles have been added (models.py, roles.py)
Groups have been removed (models.py, groups.py)
longest_token_life replaced with longest_token_life_minutes in all occurences

admin.py

/tokens endpoint commented out (need to remove properly)

groups.py

Removed

main.py

Added an audit logger (check if I'm using this)
Removed the groups blueprint
Added '/roles' as the prefix for the interface to roles.py

models.py

Added roles class + relationships (see diagram)
Token renamed to TokenHistory (may not take this)
Edited the User class to use the Role latest and longest in validity check
Changed Server from having Groups and Capabilities to Roles and Scopes
Removed all GroupServer classes
Changed make_demodata to use a limited set of Scopes and Users (may expand back to full?)
Replaced 'owner' with 'user' and 'expires' with 'expiry'
(Double check relationship cascade rules a: work and b: are appropriate)

roles.py

Created
Implements endpoints for roles editing:
- listing all roles
- listing roles for a user
- getting a specific role
- creating a new role
- editing a role (should this merge scopes or replace them with the new ones?)
- deleting a role
- getting the members of a role
- getting the scopes from a role
- listing all roles for a user on a server
- listing roles on a server

server.py

Replaced 'capabilites' endpoint with 'roles' (replication from roles.py - where live?)
Added endpoint that lists scopes on a server
Added endpoint that updates scopes on a server
Added endpoint for activating and deactivating scopes on a server (need to check if I went through all the way with this or not)
Added capabilites to edit_server to change the roles of a server
(Does SQLAlchemy Cascade take care of deleting server roles?)

token_management.py

rewrote list_my_servers()
replaced 'expires' with 'expiry' in token args
Implemented new token generation (Keep the currently commented-out API side checks?)

user_settings.py

Added an endpoint that gets a set of roles for the logged-in user

users.py

Stripped out references to groups
Added 'roles' to edit_user endpoint

Flowauth frontend changes

AdminMenu.jsx -Groups replaced with Roles

Dashboard.jsx

serverName removed for UserServer
group_admin removed
role_admin added

GroupDetails.jsx

Removed

Picker.jsx

Sanctioned abuse of Javascript functional programming to return the right value

RoleDetails.jsx

Created
View for creating and editing roles
Used to define the following:
- Role name
- Server role is associated with
- Users with a role (capability shared with User view)
- Expiry date of the role
- Scopes granted by a role
Implemented in the functional React style

GroupList.jsx/RoleList.jsx

Renamed GroupList to RoleList
Changed the Lister from GroupDetails to RoleDetails

GroupMembersPicker.jsx/RoleMembersPicker.jsx

Renamed GroupMemebersPicker to RoleMembersPicker
Replaced getGroupMembers with getRoleMembers

RoleScopePicker.jsx

Created
Component for selecting scopes from a tree into a role
Based on and similar to the existing ServerScopePicker
Takes as properties a role_id and a server_id and an updateScopes callback
Presents every scope available to server_id as an item in a CascadePicker
Presents any role already available to role_id as already checked
On update, passes new values to updateScopes
There are extensive comments on the details of this, would encourage reviewer to read

ScopeDetails.jsx

(Don't think I actually used this in the end, so can maybe remove?)

ServerAdminDetails.jsx

Removed references to capabilites
Added references to Scopes
Updated scopeGraph signature
(This could do with some renaming of variables to make processes clearer)
(Also lots of console.log() to remove)

TokenBuilder.jsx

Created
View used to make new tokens for a user by selecting their roles
Only view reachable from a non-admin account
Takes as properties a server and an onClick callback, usually used to return to prev. menu
Used to define the following:
- Token name (validated on update)
- Roles to be included in token (why would a user not click checkAll?)
On successful submit, returns user to the server token management view, where the new token is visible for copy, view or download.

TokenDetails

Rewritten to use roles + scopes
Now deirves expiry from role that is furthest in the future (this should prob. be other way around)

TokenList.jsx

Added error log

TokensRolePicker

Created
Component for listing the roles available to build a token
Takes as properties a list of roles, a checkAll function, a handleToggle function and a 'checked' list of checked items

UserAdminDetails.jsx

Removed groups
Refactored createUser + editUser to use new API endpoint

UserRoleList.jsx

Created
Component used in TokenBuilder to provide a list of roles from a user
Takes as properties:
- a list of role objects
- a checkAll function
- a handleToggle callback
- a 'checked' list containing the indicies of checked items in the list of role objects
- a setRoleState callback (I think this is unused and can be removed)

UserRolePicker.jsx

Created
Component used in UserAdminDetails to pick roles for a given user
Properties:
- user_id - the ID of the user in question
- updateRoles - a callback to call when the selected roles are changed

UserServer.jsx

Replaced TokenDetails with TokenBuilder view
(Can prob delete TokenDetails view later, but want to use it to model tests on for now)

api.js

Added 'roles' arg to editUser endpoint call
Removed group-related functions
Either removed capability-related functions or replaced 'capability' with 'scopes' or 'roles' as appropriate
Replaced 'longest_token_life' with 'longest_token_life_minutes'
Added endpoint calls for:
- getting server scopes
- getting all roles
- getting specific roles
- Creating a new role (returns the new role)
- Deleting a role
- getting roles for a user
- getting roles for a user on a server
- editing a role in it's entireity (editRole)
- renaming a role (I don't think this is used and it can be removed)
- editing role members
- editing role scopes
  - All role edits call the same backend endpoint (PATCH '.../roles/) with a varying json payload
  - (I'm not sure that any apart from editRole are actually used, though)
Changed the createToken endpoint call to use roles instead of claims

util.js

Almost full rewrite; most of the functions in here are scope-tree mangling of some form
Rewrote jsonify to use the new scopes function
Added docstrings to jsonify
Added a scopesGraph function that turns a list of scopes into a nested tree
Impleneted a function for finding the highest common roots of two trees, used in RoleScopePicker
Implemented a function for rotating a list of scopes_with_roles to a list of scopes-from- role, again used in the picker functions.

Still to do:

Integrate James' PR
Take James' comment on the poll query into consideration
Check simple permissions work
Check inside add_role for duplicate roles
Check expiry rules for tokens
Unit tests for all new components
Docstrings for new React components, paying special attention to component properties
Check for and remove unneded bits from frontend
Strip out many debug logs from frontend work

Aug 16 '22 10:08 Thingus

Can we have a quick sequence diagram to show how the auth flow works now?

Sep 02 '22 13:09 greenape

Add Token save dialog permits blank token names

Sep 16 '22 10:09 Thingus

New diagram in PR text

Oct 10 '22 09:10 Thingus

Codecov Report

Merging #5163 (cf93b8d) into master (3678d82) will decrease coverage by 0.50%. The diff coverage is 88.04%.

@@            Coverage Diff             @@
##           master    #5163      +/-   ##
==========================================
- Coverage   93.72%   93.21%   -0.51%     
==========================================
  Files         277      277              
  Lines       10852    10811      -41     
  Branches      919      894      -25     
==========================================
- Hits        10171    10078      -93     
- Misses        552      604      +52     
  Partials      129      129

Impacted Files	Coverage Δ
flowapi/flowapi/api_spec.py	`91.89% <ø> (ø)`
flowapi/flowapi/config.py	`86.66% <ø> (ø)`
flowapi/flowapi/jwt_auth_callbacks.py	`100.00% <ø> (ø)`
flowauth/backend/flowauth/util.py	`100.00% <ø> (ø)`
flowkit_jwt_generator/flowkit_jwt_generator/cli.py	`100.00% <ø> (ø)`
flowkit_jwt_generator/flowkit_jwt_generator/jwt.py	`98.01% <33.33%> (-1.99%)`	:arrow_down:
flowauth/backend/flowauth/user_settings.py	`90.69% <57.14%> (-1.93%)`	:arrow_down:
flowauth/backend/flowauth/roles.py	`79.67% <79.67%> (ø)`
flowauth/backend/flowauth/servers.py	`89.43% <80.00%> (-10.57%)`	:arrow_down:
flowauth/backend/flowauth/users.py	`96.00% <83.33%> (-2.08%)`	:arrow_down:
... and 17 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Dec 16 '22 15:12 codecov[bot]

Permissions rework

I have:

Description

Test summary

Run details

Overarching changes

FlowAPI changes

Flowauth backend changes

Flowauth frontend changes

Codecov Report