HomeUniteUs
HomeUniteUs copied to clipboard
hackforla
Implement Guest Workflow Tasks Endpoints
Overview
The API shall manage the HUU housing Workflow. API clients (e.g. the front-end application) should be able to view Tasks (name, description, status) assigned to Guests by the Workflow. The API clients should also allow Guests to navigate to Tasks to view and work on.
Tasks are an abstract concept and their concrete implementations represent such things as Forms, Training/Lessons, Scheduling, Matching, etc. The Tasks in the Workflow are logically grouped together into Task Groups. The Task Groups represent major sub-processes of the Workflow such as "Application & Onboarding process", "Host Matching process", "Match Finalization process", etc.
For this issue, all Task Groups and Tasks up until the matching process should be returned by the API.
Task Groups and Tasks go through phases represented by the following statuses:
- Locked
- Start
- In progress
- More information is needed
- Complete
The API shall maintain the following invariants imposed by the Workflow:
- The order of the Task Groups represents the order in which the Guest must complete the Task Groups.
- The order of the Tasks represents the order in which the Guest must complete the Tasks within a Task Group.
- The Tasks in a Task Group can not be unlocked until the Task Group is unlocked and the Task before it has been completed. If it's the first Task in the Task Group, then it is automatically unlocked only if the Task Group is unlocked.
- A Task Group is completed only when all of its Tasks are completed. The next Task Group can then be unlocked.
Action Items
Domain concerns:
- [ ] Implement object model for Guests, Task Groups, Tasks, Workflow and their associations
- [ ] Implement state machine representing the (Guest) Workflow
The following data must be available in the endpoint(s):
- [ ] Guest Task Groups and their Tasks up until the matching process.
- [ ] Task Group ID, Title, Status, and Tasks.
- [ ] Task ID, Title, Description, Status, Link Text, Link URL Path
Database concerns:
- [ ] Implement a persistence layer retrieve, store, and update Tasks
- [ ] Remove unused tables
- [ ] Implement relational model for Task Groups, Tasks, Workflow and associations with Guests
- [ ] Implement migration script(s)
Resources/Instructions
- Section 2 Epic
- Section 2 Guest Epic Google Doc
- Frontend Work
- Frontend Designs
- Workflow on Figma
- State machine issue
Overview
Create the endpoints needed by the frontend application to create and save the guest tasks. The guest onboarding process requires the guest to complete multiple tasks. Each task will have some associated data that needs to be stored. The onboarding is considered complete once all tasks are complete.
The backend should store the overall progress of the guest application, along with the detailed information associated with each task.
Requirements
In this initial PR, we will only store completed tasks. We can introduce draft task persistance in the future, this PR will be complex enough without this additional feature.
- Store the guest's main task progress in the HUU database
- Store the information associated with each guest task
- Add functionality needed to delete the guest tasks
Research Questions
List the steps that we need to store in the database
-
application_and_onboarding -
host_matching -
host_match_finalization
List the data we need to store for each task
Great research! If possible, we should try to avoid representing these very detailed questions and answers as concrete data models. An "easy" approach would be to define a separate model for each application and add an entry for each user. Doing this, however, would be cumbersome and brittle. Adding/Removing questions would require altering our data model, and adding new applications would require creating new models. Updates to the underlying model are expensive because each time you update a database model you need to migrate previous versions. See the research sections I added below for an idea on how to achieve this.
As a starting point, we will take questions from this application https://docs.google.com/forms/d/e/1FAIpQLSc2Zm709r_7avFQYcaL9pZRwAnUknCZengn8rXP6jxx3sm9vQ/viewform?gxids=7757.
https://www.figma.com/file/BNWqZk8SHKbtN1nw8BB7VM/HUU-Everything-Figma-Nov-2022?type=design&node-id=9669-3122&mode=design&t=o5UXHReGcRoipBRO-0
Database
Do user accounts have roles associated with them?
No. This is a problem because we need to know if the user is a guest, a host, an admin, etc. to be able to dictate what actions they can take on the webapp, i.e. what endpoints they are allowed to access. This could also be a privacy concern because we need to make sure guests and hosts can only access their own data.
How do you define a new data model?
# HUU API
# /api/openapi_server/models/database.py
from sqlalchemy import create_engine
from sqlalchemy.engine import Engine
from sqlalchemy.orm import Session
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String, ForeignKey, DateTime, LargeBinary, Boolean
from os import environ as env
DATABASE_URL = env.get('DATABASE_URL')
Base = declarative_base()
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True, index=True)
first_name = Column(String(80))
last_name = Column(String(80))
email = Column(String, unique=True, index=True)
email_confirmed = Column(Boolean, default=False)
password_hash = Column(String)
date_created = Column(DateTime)
is_admin = Column(Boolean, default=False)
is_host = Column(Boolean, default=False)
is_guest = Column(Boolean, default=False)
# optional decriptive string representation
def __repr__(self):
return f"<User(id={self.id}, first_name={self.first_name}, last_name={self.last_name}, email={self.email}, date_created={self.date_created}, is_admin={self.is_admin}, is_host={self.is_host}, is_guest={self.is_guest})>"
What are data models used for?
- SQLAlchemy models are OOP Python representations of the database tables.
- The database (a postgreSQL instance) uses the data model from SQLAlchemy to create the tables and columns in the database.
# creating tables from models in db
from sqlalchemy import create_engine
engine = create_engine("sqlite://", echo=True, future=True)
Base.metadata.create_all(engine)
How do you use the new data model to actually add an entry to the database?
- Review: Frontend sends data to backend API via HTTP POST request. Business logic validates data and uses repository to add data to database.
- Sample Data send from business logic to add to database
{
"first_name": "Alejandro",
"last_name": "Gomez",
"email": "[email protected]",
"email_confirmed": false,
"password_hash": "lfOcifi3DoKdjfvhwlrbugvywe3495!#$%",
"date_created": "2023-09-19 12:00:00",
"is_admin": false,
"is_host": false,
"is_guest": true
}
# api/openapi_server/repository/guest_repository.py
from typing import Optional, List
from sqlalchemy.orm import Session
from openapi_server.models.database import User, DataAccessLayer
"""
inserting data into the database
"""
class GuestRepository:
def create_guest(self, data: dict) -> Optional[User]:
"""
create a new guest - adds data entry to the database
"""
with DataAccessLayer.session() as session:
new_guest = User(
# db auto generates and auto increments an id
first_name=data["first_name"],
last_name=data["last_name"],
email=data["email"],
email_confirmed=data["email_confirmed"],
password_hash=data["password_hash"],
date_created=data["date_created"],
is_admin=data["is_admin"],
is_host=data["is_host"],
is_guest=data["is_guest"]
)
session.add(new_guest)# places instance into the session
session.commit() # writes changes to db
session.refresh(new_guest) # erases all attributes of the instance and refreshes them with the current state of the db by emitting a SQL query. this is important for autoincrementing id
return new_guest # returns the info from the db to the business logic
How do you retrieve data models from the database
- Above, the repository would return the new_guest data model to the business logic with its unique id. Using this unique id, we can use a different business logic to get the data model from the database using the repository.
# api/openapi_server/repository/guest_dashboard_repository.py
from typing import Optional, List
from sqlalchemy.orm import Session
from openapi_server.models.database import User, DataAccessLayer
class GuestRepository:
def get_guest_by_id(self, id: int) -> Optional[User]:
"""
gets a guest by id
"""
with DataAccessLayer.session() as session:
return session.query(User).get(id)
How do you define a relationship between two tables in a database using SQLAlchemy
Good example showing a reference, but the terminology 'task' and 'subtask' is a bit misleading. A task has a specific meaning in asynchronous programming. It is typically a wrapper around a function that you want to run asynchronously.
I'm thinking instead of Task and Subtask, we can have a Dashboard, Application, & Questions. A dashboard has several ordered applications, with a progress indicator for each item.
Addressed further in a comment below. We are using task_group -> tasks
# HUU API
# /api/openapi_server/models/database.py
from sqlalchemy import create_engine
from sqlalchemy.engine import Engine
from sqlalchemy.orm import Session, relationship
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String, ForeignKey, DateTime, LargeBinary, Boolean
from os import environ as env
DATABASE_URL = env.get('DATABASE_URL')
Base = declarative_base()
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True, index=True)
first_name = Column(String(80))
last_name = Column(String(80))
email = Column(String, unique=True, index=True)
email_confirmed = Column(Boolean, default=False)
password_hash = Column(String)
date_created = Column(DateTime)
is_admin = Column(Boolean, default=False)
is_host = Column(Boolean, default=False)
is_guest = Column(Boolean, default=False)
tasks = relationship("Task", backref="user")
class Task(Base):
__tablename__ = 'task'
id = Column(Integer, primary_key=True, index=True)
guest_id = Column(Integer, ForeignKey("user.id"))
title = Column(String(80))
status = Column(String(80))
subtasks = relationship("Subtask", backref="task")
class Subtask(Base):
__tablename__ = 'subtask'
id = Column(Integer, primary_key=True, index=True)
guest_id = Column(Integer, ForeignKey("user.id"))
task_id = Column(Integer, ForeignKey("task.id"))
status = Column(String(80))
How do you convert a data model into JSON?
- We encode the in-memory data (or serialize/marshall) before we transport it so other programs can convert it to their own in-memory data; i.e. universal data language. On HUU, we are using Marshmallow to serialize.
- HUU Schema with Marshmallow
- HUU requirements.txt - Marshmallow
- Marshmallow docs
Endpoints
Where are endpoints located within the API code?
- Our endpoints are in the
openapi.yamlfile - {URL}/api/auth/signin
-
# HomeUniteUs/api/openapi_server/openapi/openapi.yaml post: # POST Request description: Sign in a user operationId: signin requestBody: # email and password as a string enters endpoint content: application/json: schema: type: object properties: email: type: string password: type: string required: - email - password responses: '200': content: application/json: schema: # response comes from the schema... shown below $ref: '../../openapi.yaml#/components/schemas/ApiResponse' description: successful operation tags: - auth x-openapi-router-controller: openapi_server.controllers.auth_controller ####################################################################### # New file below # ####################################################################### # HomeUniteUs/api/openapi_server/openapi/openapi.yaml # **endpoints and rest of yaml doc goes here...** components: securitySchemes: jwt: type: http scheme: bearer bearerFormat: JWT x-bearerInfoFunc: openapi_server.controllers.security_controller.requires_auth responses: $ref: "./responses/_index.yaml" parameters: $ref: "./parameters/_index.yaml" schemas: $ref: "./schemas/_index.yaml" ApiResponse: # according to this, the response should be a code, type, and message example: code: 0 type: type message: message properties: code: format: int32 title: code type: integer type: title: type type: string message: title: message type: string title: ApiResponse type: object- authSignin.yaml
- I think our openAPI needs some tweaking - the auth/signin controller returns an object
{ "token" : access_toke, "user" : user}which doesn't fit the schema above
Describe the process for adding a new endpoint
-
Step 1: Add the api route, i.e. the endpoint
-
# HomeUniteUs/api/openapi_server/openapi/openapi.yaml openapi: 3.0.0 info: license: name: GPL 2.0 title: Home Unite Us version: 1.0.0 servers: - url: http://homeunite.us/api paths: /users/{user_email}: # added path $ref: "./paths/usersEmail.yaml" # rest of paths below...
-
-
Step 2: Add the user email parameter
-
# HomeUniteUs/api/openapi_server/openapi/parameters/_index.yaml Email: name: user_email description: The email of the user to retrieve in: path required: true schema: type: string style: simple
-
-
Step 3: Add the user response schema
-
# HomeUniteUs/api/openapi_server/openapi/schemas/_index.yaml ApiResponse: example: code: 0 type: type message: message properties: code: format: int32 title: code type: integer type: title: type type: string message: title: message type: string title: ApiResponse type: object User: type: object properties: id: format: int64 type: integer first_name: title: first_name type: string last_name: title: last_name type: string email: title: email type: string email_confirmed: title: email_confirmed type: boolean password_hash: title: password type: string date_created: title: date_created type: string format: date-time is_admin: title: is_admin type: boolean is_host: title: is_host type: boolean is_guest: title: is_guest type: boolean required: - id - first_name - last_name - email - email_confirmed - date_created - is_admin - is_host - is_guest
-
-
Step 4: Add endpoint path info
-
# HomeUniteUs/api/openapi_server/openapi/paths/usersEmail.yaml get: description: Get user by email operationId: get_user_by_email # this is the controller function tags: - users parameters: - $ref: "../parameters/_index.yaml#/Email" responses: "200": description: Successful response content: application/json: schema: $ref: "../schemas/_index.yaml#/User" default: description: Unexpected error $ref: "../responses/_index.yaml#/UnexpectedError" x-openapi-router-controller: openapi_server.controllers.users_controller
-
-
On the HUU Repo
- Link to commit that encompasses code to retrieve a user by email
Testing
How do you run the backend test cases?
- Start up the virtual environment. It looks like our development venv is shared with testing so I'll use that one.
- run
pytestfrom the 'api' directory and all tests in the 'test' directory will run. It will display=================== 27 passed, 209 warnings in 8.30s ===================
What is a pytest fixture?
Fixtures are functions that typically provide resources for the tests.
- preparing objects/data
- starting/killing services
- entering records to db
- etc.
- pytest docs
import pytest
def create_user(first_name, last_name):
return {"first_name": first_name, "last_name": last_name}
@pytest.fixture
def new_user():
return create_user("Alejandro", "Gomez")
@pytest.fixture
def users(new_user):
return [create_user("Jose", "Garcia"), create_user("Juan", "Lopez"), new_user]
def test_new_user_in_users(new_user, users):
assert new_user in users
Might not be important in this issue, but it is important to note that pytest fixtures also allow you to perform setup and teardown. You can do this using the
yieldkeyword.
Thanks for showing this! This will be super useful
@pytest.fixture
def demo_setup_teardown():
# Code before yield is executed before test starts
doSetup()
yield testData
# Code after yield is executed after test finishes
# Even if an exception is encountered
doTeardown()
Show a test case that adds a value to the database and checks it
# HomeUniteUs/api/openapi_server/test/test_service_provider_repository.py
# Third Party
import pytest
# Local
from openapi_server.models.database import DataAccessLayer
from openapi_server.repositories.service_provider_repository import HousingProviderRepository
@pytest.fixture
def empty_housing_repo() -> HousingProviderRepository:
'''
SetUp and TearDown an empty housing repository for
testing purposes.
'''
DataAccessLayer._engine = None
DataAccessLayer._conn_string = "sqlite:///:memory:"
DataAccessLayer.db_init()
yield HousingProviderRepository()
test_engine, DataAccessLayer._engine = DataAccessLayer._engine, None
test_engine.dispose()
@pytest.fixture
def housing_repo_5_entries(empty_housing_repo: HousingProviderRepository) -> HousingProviderRepository:
'''
SetUp and TearDown a housing repository with five service providers.
The providers will have ids [1-5] and names Provider 1...Provider5
'''
for i in range(1, 6):
new = empty_housing_repo.create_service_provider(f"Provider {i}")
assert new is not None, f"Test Setup Failure! Failed to create provider {i}"
assert new.id == i, "The test ids are expected to go from 1-5"
yield empty_housing_repo
# this function adds a value to the db and checks it
def test_create_provider(empty_housing_repo: HousingProviderRepository):
'''
Test creating a new provider within an empty database.
'''
EXPECTED_NAME = "MyFancyProvider"
newProvider = empty_housing_repo.create_service_provider(EXPECTED_NAME) # adds value to the db via a pytest.fixture of a db and HousingProviderRepository
# checks the value returned to assert a test status
assert newProvider is not None, "Repo create method failed"
assert newProvider.id == 1, "Expected id 1 since this is the first created provider"
assert newProvider.provider_name == EXPECTED_NAME, "Created provider name did not match request"
Show a test case that tests an endpoint
# HomeUniteUs/api/openapi_server/test/test_service_provider_controller.py
from __future__ import absolute_import
from openapi_server.test import BaseTestCase
class TestServiceProviderController(BaseTestCase):
"""ServiceProviderController integration test stubs"""
def test_create_service_provider(self):
"""
Test creating a new service provider using a
simulated post request. Verify that the
response is correct, and that the app
database was properly updated.
"""
REQUESTED_PROVIDER = {
"provider_name" : "-123ASCII&" # creates a fake test object
}
response = self.client.post(
'/api/serviceProviders',
json=REQUESTED_PROVIDER) # sends POST request with payload to API endpoint
self.assertStatus(response, 201, # asserts the response status
f'Response body is: {response.json}')
assert 'provider_name' in response.json # asserts data in response object
assert 'id' in response.json
assert response.json['provider_name'] == REQUESTED_PROVIDER['provider_name']
# verifies there was a 'write' on the db, i.e. that the db was updated based on the API endpoint response
db_entry = self.provider_repo.get_service_provider_by_id(response.json['id'])
assert db_entry is not None, "Request succeeeded but the database was not updated!"
assert db_entry.provider_name == REQUESTED_PROVIDER['provider_name']
Data Model Questions
Outline the relationships between Guests, Providers, Coordinators, Hosts and applications
We hope to support multiple providers. Each provider will have their own set of requirements. Ideally providers would be able to add new questions, and custom tailor their guest and host applications.
- A guest will be serviced by exactly one provider.
- A host will be serviced by exactly one provider.
- A guest will have exactly one coordinator
- A host will have exactly one coordinator
- The coordinator will be associated with a provider (the same as the guest / host provider)
- A single provider will have many guests, hosts, and coordinators
- HUU will service multiple providers
- Each provider will have a different set of guest & host applications
How should we design our database model?
Our design should support unique applications for each provider. If we create explicit database models for each application, then we will need to modify our database model each time an application is edited or each time a new application is added.
We can avoid restructuring our entire database model each time a routine application modification is made by abstracting the application requirements and constructing the provider applications at runtime.
To achieve this, we could store each application as table. Applications would contain a list of ordered questions. Adding a question would require adding a row to an existing application table. Adding a new application would require adding a new application table. In both cases the underlying schema would remain the same.
What does our current data model look like?
The ER diagram was generated using the ERAlchemy package. What's interesting is that most of these models are not used by our application at all.
What should our data model look?
This design supports provider-specific applications, and would allow us to add/remove/edit applications without editing the data model.
The provider_id & role define the dashboard that the frontend will use. The dashboard defines the collection of applications that need to be completed, along with each application's progress. The application defines the set of questions that need to be asked. Each question defines the question's text and the expected response type.
Responses to the User's question are stored in the Response table. We can use this table to easily look up user responses for a given application, using the (user_id, question_id) composite key. If no response is found then we know that application has an unanswered question.
---
title: HUU Application Dashboard Data Model
---
erDiagram
User {
int user_id PK
int provider_id FK
string name
string email UK
enum role "Guest,Host,Coord"
}
Provider {
int provider_id PK
string name
}
Dashboard {
int dashboard_id PK
int provider_id FK
string title
enum role "Provider has guest & host dashboard"
}
Application {
int app_id PK
int dashboard_id FK
int dashboard_order
string title
enum progress
}
App_Question_Junction {
int app_id FK
int question_id FK
int question_order
}
Question {
int question_id PK
int text
int response_type_id FK
}
ResponseType {
int response_type_id PK
string type_name "str, bool, int, etc"
}
Response {
int user_id FK
int question_id FK
string response_value
date timestamp
}
Provider }|--|| User : has
Provider ||--|{ Dashboard : defines
Dashboard ||--|{ Application :"consists of"
Application }|--o{ Question : "utilizes"
ResponseType ||--o{ Question : "defines"
User }o--o{ Response : "user answers"
Current API Questions
Show the code that is used to sign up new Guests
Do we do anything to identify Guest user accounts as 'Guest', as opposed to 'Host', 'Coordinator', 'etc'? The Applicant model provides a mechanism for associating a user with a role. It doesn't look like we use this model yet.
I think we should consider restarting our models as you suggested. Or at minimum get together and clean up (remove) unused models.
We have a model that handles the identification of user, i.e. applicant type. Here's the models used for signing up a user
# HomeUniteUs/api/openapi_server/models/database.py
class User(Base):
__tablename__ = "user"
id = Column(Integer, primary_key=True, index=True)
email = Column(String, nullable=False, unique=True)
class ApplicantType(Base):
__tablename__ = "applicant_type"
id = Column(Integer, primary_key=True, index=True)
applicant_type_description = Column(String, nullable=False)
class ApplicantStatus(Base):
__tablename__ = "applicant_status"
id = Column(Integer, primary_key=True, index=True)
applicant_type = Column(Integer, ForeignKey('applicant_type.id'), nullable=False)
status_description = Column(String, nullable=False)
class Applicant(Base):
__tablename__ = "applicant"
id = Column(Integer, primary_key=True, index=True)
applicant_type = Column(Integer, ForeignKey('applicant_type.id'), nullable=False)
applicant_status = Column(Integer, ForeignKey('applicant_status.id'), nullable=False)
user = Column(Integer, ForeignKey('user.id'), nullable=False)
Here is what the code could like if we wanted to implement a new guest user, indicating their role
# HomeUniteUs/api/openapi_server/controllers/auth_controller.py
from openapi_server.models.database import DataAccessLayer, User, ApplicantType, ApplicantStatus, Applicant
def signUpGuest(): # noqa: E501
"""Signup a new Guest
"""
if connexion.request.is_json:
body = connexion.request.get_json()
secret_hash = get_secret_hash(body['email'])
# Signup user as guest
with DataAccessLayer.session() as session:
user = User(email=body['email'])
applicant_type_id = session.query(ApplicantType.id).filter_by(applicant_type_description="guest").first()
applicant_status = ApplicantStatus(
applicant_type=applicant_type_id,
status_description="unconfirmed_email"
)
session.add_all([user, applicant_status])
# commit and refresh to db to be able to get applicant_status.id and user.id as they will be autogenerated
session.commit()
session.refresh(user, applicant_status)
applicant = Applicant(
applicant_type=applicant_type_id,
applicant_status=applicant_status.id,
user = user.id
)
session.add(applicant)
try:
session.commit()
except IntegrityError:
session.rollback()
raise AuthError({
"message": "A user with this email already exists."
}, 422)
try:
response = userClient.sign_up(
ClientId=COGNITO_CLIENT_ID,
SecretHash=secret_hash,
Username=body['email'],
Password=body['password'],
ClientMetadata={
'url': ROOT_URL
}
)
return response
except botocore.exceptions.ClientError as error:
match error.response['Error']['Code']:
case 'UsernameExistsException':
msg = "A user with this email already exists."
raise AuthError({ "message": msg }, 400)
case 'NotAuthorizedException':
msg = "User is already confirmed."
raise AuthError({ "message": msg }, 400)
case 'InvalidPasswordException':
msg = "Password did not conform with policy"
raise AuthError({ "message": msg }, 400)
case 'TooManyRequestsException':
msg = "Too many requests made. Please wait before trying again."
raise AuthError({ "message": msg }, 400)
case _:
msg = "An unexpected error occurred."
raise AuthError({ "message": msg }, 400)
except botocore.excepts.ParameterValidationError as error:
msg = f"The parameters you provided are incorrect: {error}"
raise AuthError({"message": msg}, 500)
How should the frontend and backend interact?
The plan is to implement endpoints that the frontend can use to query a user-specific dashboard populated with applications & their progress. The dashboard will contain app_ids that can be used to load and save the application in the backend. Each application will contain an ordered list of questions and responses
---
title: Application Logic Flow
---
flowchart TB;
subgraph frontend;
f1["Client\nSign in"]
f2["Load page\nusing role"]
f3["Show Dashboard\n & Wait"]
f4["Dashboard\nApp OnClick"]
f5["Display\nApp"]
f6["App\nOnSave"]
f1 ~~~ f2 ~~~ f3 ~~~ f4 ~~~ f5 ~~~ f6
end
subgraph backend;
b1["sign-in()"]
b2["app_dashboard()"]
b3["get_app"]
b4["update_app()"]
b1 ~~~ b2 ~~~ b3 ~~~ b4
end
f1 --"POST\n{\nusername,\n password\n}"--> b1
b1 --"{\n role\n jwt\n}"--> f2
f2 --"GET\n/api/dashboard"--> b2
b2 --"{\napp1 {\nprogress,\ntext,\napp_id\m},\napp...\n}"--> f3
f4 --"GET\n/api/application/app_id"--> b3
b3 --"{\nprogress\nquestion1{\n progress,\n text,\n response_type,\n response\n},\nquestion2...\n}"--> f5
f6 --"PUT\n{\nprogress\nquestion1{\n progress,\n text,\n response_type,\n response\n},\nquestion2...\n}"--> b4
linkStyle 8,9,10,11,12,13,14 text-align:left
General Solution Plan
Implement a flexible application model backend system, that will allow the front-end app to query Guest and Host applications using a user_id. See the model erdiagram and application flow diagram above.
Each user will be associated with a single provider. Each provider will have a unique guest and host application. The front-end application will receive all the information it needs to dynamically generate guest and host applications.
Our backend has a 'demo' database model, however many of the models are currently unused by the frontend application. We need to decide if we want to start fresh, or if want to modify the existing model to meet our current needs. Modifying an unused model can be very challenging since we do not know if the current unused model works. I propose removing the unused models and enforcing a requirement that all new models need to be used by either the frontend app or our test project.
Model
We will implement the model by starting at the user and working our way towards responses. We will create test cases to exercise the new models at each step.
- Introduce a
Roleenumerated type withGuest,Host,Coordinator, andAdminroles - Update the
Usermodel to include a role and provider_id - Update the
signupmethods to specify a role when the user signs up - Add a
ResponseTypemodel that defines the supported response types. The frontend will rely on this to interpret and edit the question responses - Add a
Questionmodel to store the bank of questions. The question bank will be shared across applications. - Add a
Dashboardmodel to store a provider's role-based dashboards. - Add a
Applicationmodel to store the dashboard's ordered list of applications - Add a
AppQuestionJunctiontable to store the many-to-many relationship between each application and the ordered list of questions
Database Migration
With these changes we will need to store the dashboard and application structure within the database, since this structure can be defined and edited by each provider.
- Write a script that can create a development database. This development database should contain:
- A default provider
- A default Guest Dashboard
- A default Host Dashboard
- Applications & Questions used by the dashboards
- Write a migration script that can update our existing database to use the new model.
- Remove all unused models to simplify future development
Endpoint
- Add a
app_dashboard()endpoint- Retrieve the dashboard using a user_id and provider_id
- Return a dashboard as json, containing an ordered list of applications. Each application will include a progress field and an
app_idthat can be used to query the application
- Add a
get_app()endpoint- Retrieve the application using the app_id
- Populate the response for each question by querying the
Responsetable using theuser_idandquestion_id. If no response is found then the user has not answered that question. If a response is found, then return it within the application.
- Add a
update_app()endpoint-
PUTthe same basic json received from theget_app()endpoint, except theresponse_valuefields will contain user responses
-
Frontend
This will be apart of a separate issue.
Implementation Questions
Hey @agosmou,
An initial project plan is ready for this issue. Please submit your answers and reassign me once it is ready for review.
My comments are included as
quoted text
Please don't remove those section. I'll remove them as I review your responses.
Working through this on markdown doc in VSCode. Ill paste it in here when I finish over the next day or so.
@agosmou @Joshua-Douglas I simplified the dashboard data a bit and wanted to leave an example here for reference incase it's helpful.
Sample response for tasks and sub-tasks:
{
id: 1,
title: 'Application and Onboarding',
status: 'in-progress',
subTasks: [
{
id: 1,
title: 'Application',
status: 'complete',
description:
'Start your guest application to move on to the next step.',
buttonText: 'Start application',
url: '/guest-application',
},
{
id: 2,
title: 'Coordinator Interview',
status: 'in-progress',
description: 'Meet with your Coordinator to share more about yourself.',
buttonText: 'Schedule interview',
url: '/schedule',
},
{
id: 3,
title: 'Training Session',
status: 'locked',
description:
'Complete a training session to prepare you for the host home experience.',
buttonText: 'Schedule training',
url: '/schedule',
},
],
},
Potential entity relation diagram:
erDiagram
USER ||--|{ TASK : has
TASK {
int id PK
int userId FK
string title
int statusId FK
}
TASK ||--|{ SUB_TASK: contains
TASK ||--|| STATUS: has
SUB_TASK ||--|| STATUS: has
SUB_TASK{
int id PK
int taskId FK
string title
string description
int statusId FK
string buttonText
url string
}
STATUS {
int id PK
string status "locked, inProgress, complete"
}
@erikguntner
Awesome info. Thanks Eric! I'm wrapping up the design doc. I'll post this afternoon.
@agosmou this looks awesome! Just FYI Cognito tracks whether the user is confirmed in case that changes whether we want to keep that info in the database as well or not
Hey @agosmou, Thanks for the designs! I'll make sure to review them by Wednesday night!
Figma Designs for backend-design consideration
@Joshua-Douglas - I can review these tomorrow night to see if there are any effects on the above
https://www.figma.com/file/BNWqZk8SHKbtN1nw8BB7VM/HUU-Everything-Figma-Nov-2022?type=design&node-id=9669-3122&mode=design&t=oPs2UK9Ee4GFVwK4-0
Hey @erikguntner,
Thanks a lot for the Task/SubTask idea. After reading through @agosmou's research I think a generic approach like this is going to be easier to implement than the more 'naive' approach of defining each application as a separate model.
It would be great if you could look through the proposed data model & application flowchart I posted above. If the frontend is already relying on the backend to query the dashboard and applications then that gives us the flexibility to define provider-specific and role-specific dashboards. This would mean that each individual provider could define a custom Guest and Host dashboards, complete with custom applications.
Hey @agosmou,
The design document is ready for review! Your research looked great, and it inspired me to think up a generic approach that could accommodate the Task/Subtask approach outlined by @erikguntner.
I left
comments
on some of your responses, and added several new sections (everything after the Data Model Questions section). I left one question un-answered, so I could really use your help getting to the bottom of it.
Can you answer that question, and review the design? I'd like to get your feedback. I'm sure something is missing or could be improved. If you agree with the general approach then I'll create a set of implementation questions that we can use to outline the trickiest parts of the implementation before opening a branch.
@agosmou The Guest Dashboard story #500 uses Dashboard -> Steps -> Tasks language to describe the Guest Dashboard (see the "Key Decisions" section). In the diagram provided by @erikguntner's, Task should be "Step" and Sub_Task should be "Task" to align with the story's description of the Guest Dashboard feature. If anyone has found it makes more sense to name these entities differently, loop @sanya301 in to iterate on the language we'll use to describe and implement the feature.
The Dashboard Application Model diagram provided by @Joshua-Douglas is a little bit out of scope for this feature but still eventually necessary. It would need to be modified to associate the Application model with a Task: A Dashboard contains Steps, Steps are a group of Tasks, Task has-a/is-a:
- Application has-a/is-a Form (or a Dashboard context representation of an Application instead of the full-fledged one)
- Training
- Interview
- Match
- etc.
Is the above a correct representation of the Guest Dashboard model?
Hey @paulespinosa, How is the guest dashboard data model diagram out of scope for the guest dashboard endpoint issue? Do you think we should split the two issues and backlog this one, or do you think that changes to the data model are not strictly necessary?
@Joshua-Douglas specifically, the "HUU Application Dashboard Data Model" diagram. It contains models (Application, Question, ResponseType, Response, App_Question_Junction), that are circled in the image below, that are more representative of an application form and are great candidate models for the issues: Guest Application Form #533 and Guest application implementation #610 and beyond.
Caveat: I've been using the terminology "Steps" and "Tasks" as defined in the main issue Guest Dashboard #500. However, it appears terminology has evolved from there, so I've pinged Sanya https://github.com/hackforla/HomeUniteUs/issues/500#issuecomment-1751054169 to motivate alignment on how the team describes the Dashboard features.
The Guest Dashboard #500 issue, the parent issue for this issue, Implement Guest Dashboard Endpoints #572, specifies displaying "high-level" information such as status, description, and progress about Steps. It's not asking to display the details of an application (form). When #500 talks about being able to "click into any additional page(s)", it means as a generic action where the details of those landing pages are unspecified and left for implementation in separate issues.
So the question becomes: how do you get the status and progress of a Step? Perhaps, by asking each of its Tasks for their status and progress. How do you get the status and progress of a Task representing, say, an Application Form? Consider, as an implementation option, defining Task as an interface or abstract base class. Have concrete Dashboard context representations of Tasks, such as an Application Form, contain the logic to calculate or get the status and progress. For this issue, the concrete implementations (or simply the Task for now) can return canned responses until specific issues are defined to implement their details (e.g. regarding Application Form, it can be an issue on its own or an action item for implementing the backend for Guest Application Form #533).
Unfortunately, the Action Items in this issue appears to be misaligned with its parent.
Based on the context of #500, the action items circled above should probably read:
- Steps associated with a Dashboard (or All Steps needed to complete the Matching Process)
- Tasks associated with each Step
- Description of each ~Step~ Task - Confirmed https://github.com/hackforla/HomeUniteUs/issues/500#issuecomment-1756395817
Regarding #500's Consideration: "In the future, the plan is for the guest onboarding experience for multiple organizations (SPY and beyond), where different organizations might want to modify the steps needed for a guest.". This word "modify" is under-specified and needs to be discussed with Product Management https://github.com/hackforla/HomeUniteUs/issues/500#issuecomment-1751054169.
According to this comment, we are doing the following naming convention:
"Task Groups has Tasks"
I think we can redefine the scope above to limit this issue to guest dashboard task groups and tasks, so we can remove the extra action items. We can use all the models above on issue #500 and develop them more further on their respective issues.
@Joshua-Douglas Take a look at the edits above on the design doc. Let me know if you want to meet to discuss our models further - the ERDiagram you posted puts into perspective the reorganizing that has to be done
cc: @tylerthome
Hi @paulespinosa - Are you able to look over my progress setting up models? I want to make sure Im understanding this and going in the right direction.
I tried my hand at what the models would look like so I could then work on the tests, domain objects, and domain logic.
As far as the statemachine (state chart pattern), we will have models.py and statechart.py that has the logic for this. At this time, I am planning on handrolling unless we agree to using a library.
- Option 1: python-statemachine with support for python 3.10
- Option 2: transitions with support for up to python 3.8 so Im not sure this can play nice with our api on python 3.10
models to be reviewed
##########################
# api/models/database.py #
##########################
import enum # adds python enum import
from sqlalchemy import create_engine
from sqlalchemy.engine import Engine
from sqlalchemy.orm import Session, declarative_base
from sqlalchemy import Column, Integer, String, ForeignKey, DateTime, LargeBinary, Enum # adds enum
from sqlalchemy.orm import relationship # adds imports
Base = declarative_base()
# proposed models...
class TaskStatus(enum.Enum):
LOCKED = 'Locked'
START = 'Start'
IN_PROGRESS = 'In Progress'
MORE_INFORMATION_NEEDED = 'More Information Needed'
COMPLETE = 'Complete'
class TaskGroupStatus(enum.Enum):
LOCKED = 'Locked'
START = 'Start'
IN_PROGRESS = 'In Progress'
MORE_INFORMATION_NEEDED = 'More Information Needed'
COMPLETE = 'Complete'
class TaskGroupSubProcess(enum.Enum):
LOCKED = 'Locked'
APPLICATION_AND_ONBOARDING_PROCESS = 'Application & Onboarding Process'
HOST_MATCHING_PROCESS = 'Host Matching Process'
MATCH_FINALIZATION_PROCESS = 'Match Finalization Process'
class Guest(Base):
__tablename__ = 'guest'
id = Column(Integer, primary_key=True, index=True)
email = Column(String, ForeignKey('user.email'), nullable=False)
tasks = relationship("Task", back_populates="guest", cascade="all, delete")
task_groups = relationship("TaskGroup", back_populates="guest", cascade="all, delete")
guest_workflows = relationship("GuestWorkflow", back_populates="guest", cascade="all, delete")
class Task(Base):
__tablename__ = 'task'
id = Column(Integer, primary_key=True, index=True)
title = Column(String, nullable=False)
description = Column(String, nullable=False)
task_group_id = Column(Integer, ForeignKey('task_group.id'))
status = Column(Enum(TaskStatus), default=TaskStatus.LOCKED, nullable=False) # from finite state machine (statechart pattern)
guest_id = Column(Integer, ForeignKey('guest.id', ondelete='CASCADE'))
# methods for state machine (statechart pattern)
class TaskGroup(Base):
__tablename__ = 'task_group'
id = Column(Integer, primary_key=True, index=True)
name = Column(Enum(TaskGroupSubProcess), default=TaskGroupSubProcess.LOCKED, nullable=False) # from finite state machine (statechart pattern)
guest_workflow_id = Column(Integer, ForeignKey('guest_workflow.id'))
status = Column(Enum(TaskGroupStatus), default=TaskGroupStatus.LOCKED,nullable=False) # from finite state machine (statechart pattern)
guest_id = Column(Integer, ForeignKey('guest.id', ondelete='CASCADE'))
guest = relationship("Guest", back_populates="task_groups")
tasks = relationship("Task", back_populates="task_groups")
# methods for state machine (statechart pattern)
class GuestWorkflow(Base):
__tablename__ = 'guest_workflow'
id = Column(Integer, primary_key=True, index=True)
guest_id = Column(Integer, ForeignKey('guest.id', ondelete='CASCADE'))
task_groups = relationship('TaskGroup', back_populates='guest_workflow')
guest = relationship("Guest", back_populates="guest_workflows")
# existing code..
class User(Base):
__tablename__ = "user"
id = Column(Integer, primary_key=True, index=True)
email = Column(String, nullable=False, unique=True)
class ApplicantType(Base):
__tablename__ = "applicant_type"
id = Column(Integer, primary_key=True, index=True)
applicant_type_description = Column(String, nullable=False)
class Applicant(Base):
__tablename__ = "applicant"
id = Column(Integer, primary_key=True, index=True)
applicant_type = Column(Integer, ForeignKey('applicant_type.id'), nullable=False)
applicant_status = Column(Integer, ForeignKey('applicant_status.id'), nullable=False)
user = Column(Integer, ForeignKey('user.id'), nullable=False)
# rest of models here...
class DataAccessLayer:
_engine: Engine = None
@classmethod
def db_init(cls, conn_string):
cls._engine = create_engine(conn_string, echo=True, future=True)
Base.metadata.create_all(bind=cls._engine)
@classmethod
def session(cls) -> Session:
return Session(cls._engine)
Hi @agosmou PM has been working on statuses in a Google Doc at https://docs.google.com/document/d/1mksBNqE9hc-bAW49mdHQDImkk8f92YewtwENNpqvFHc/edit. We (devs) will need to work together with PM on defining the statuses. As of this writing, it looks like the statuses in the Google Doc are "user statuses". It's not yet clear how the relation between "User status" and "Task status" will work.
I think the Guest should not contain references to its Tasks or TaskGroups; they should be obtained via the GuestWorkflow or similar. With respect to Guest holding a reference to the GuestWorkflow, that can be done or the GuestWorkflow can be obtained via another mechanism; we won't really know which suits us until we start working with the code and gain more clarity on the domain.
TaskStatus, TaskGroupStatus, TaskGroupSubProcess are ok to hard code for starters but will need to be stored in the DB.
Try hand rolling a basic state machine to get a feel for it and gain better clarity about our needs. Thank you.
cc: @tylerthome
Thanks for this info, @paulespinosa ! Given the 'Host' items also take status, it'd be good to keep this in mind to make the code reusable for other endpoints.
Below Iadjusted the models and made a quick run at a state machine
Adjusted Models
#models/database.py
# proposed models...
class TaskStatus(enum.Enum):
LOCKED = "Locked"
START = "Start"
IN_PROGRESS = "In Progress"
MORE_INFORMATION_NEEDED = "More Information Needed"
COMPLETE = "Complete"
class TaskGroupStatus(enum.Enum):
LOCKED = "Locked"
START = "Start"
IN_PROGRESS = "In Progress"
MORE_INFORMATION_NEEDED = "More Information Needed"
COMPLETE = "Complete"
class TaskGroupSubProcess(enum.Enum):
LOCKED = "Locked"
APPLICATION_AND_ONBOARDING_PROCESS = "Application & Onboarding Process"
HOST_MATCHING_PROCESS = "Host Matching Process"
MATCH_FINALIZATION_PROCESS = "Match Finalization Process"
class Guest(Base):
__tablename__ = "guest"
id = Column(Integer, primary_key=True, index=True)
email = Column(String, ForeignKey("user.email"), nullable=False)
guest_workflows = relationship(
"GuestWorkflow", back_populates="guest", cascade="all, delete"
)
class Task(Base):
__tablename__ = "task"
id = Column(Integer, primary_key=True, index=True)
title = Column(String, nullable=False)
description = Column(String, nullable=False)
task_group_id = Column(Integer, ForeignKey("task_group.id"))
status = Column(
Enum(TaskStatus), default=TaskStatus.LOCKED, nullable=False
) # from finite state machine (statechart pattern)
# methods for state machine (statechart pattern)
State Machine (State Chart Pattern)
pseudo-implementation of hand rolled state machine using state chart pattern
- states
- triggers
- transitions
# models/state_machine.py
class TaskStateMachine:
def __init__(self, task_id, session: Session):
self.task = session.query(Task).get(task_id)
self.session = session
self.state = self.task.status
def transition(self, trigger):
match trigger:
case "start":
self.state = TaskStatus.START
case "progress":
self.state = TaskStatus.IN_PROGRESS
case "need_info":
self.state = TaskStatus.MORE_INFORMATION_NEEDED
case "complete":
self.state = TaskStatus.COMPLETE
print(f"Transitioning task to {self.state}")
self.task.status = self.state
self.session.commit()
class TaskGroupStateMachine:
def __init__(self, task_group_id, session: Session):
self.task_group = session.query(TaskGroup).get(task_group_id)
self.session = session
self.state = self.task_group.status
def transition(self, trigger):
match trigger:
case "start":
self.state = TaskGroupStatus.START
case "progress":
self.state = TaskGroupStatus.IN_PROGRESS
case "need_info":
self.state = TaskGroupStatus.MORE_INFORMATION_NEEDED
case "complete":
self.state = TaskGroupStatus.COMPLETE
print(f"Transitioning task group to {self.state}")
self.task_group.status = self.state
self.session.commit()
Design
UI Design
~Ready for hand off - Section 2 guest dashboard
Entities
~Brainstorm - Sandbox
This should be ice-boxed at least until existing MVP scope is nominally complete -- this is a requirement for compliance and traceability, but not strictly required for stakeholder demo
Moved to New Issue Approval, since this task is required for dependent specs for MVP