RFC: new access control system

[mwest1066]

Your feedback is wanted! This is a tentative plan for a new access control system, to replace the current allowAccess rules. I'm writing this up for the purpose of collecting feedback so please comment on this issue! In particular, I'd love to hear whether there are use cases that people have which are not accommodated by this new system.

Goals

The main aims of this new system are:

1. Make it less confusing for instructors to configure the access control of their assessments. This has historically been a significant pain point. With the existing list of access rules it can be hard to understand which rules apply to any given student at any given time, and it can be hard to combine the existing access rules together to achieve a given effect (e.g., an exam which hides the questions after the exam, but then reveals them at a later date).
2. Assign more semantic meaning to the access control settings. For example, this will allow PrairieLearn to know the due date of an assessment, which isn't possible under the existing system. This will be helpful for displaying a calendar, sending reminders, etc.

The proposed new system is less flexible than the existing system, but hopefully it covers all the practical use cases that people have and is much easier to use and much harder to get wrong by accident.

Basic structure

We'll add a new accessControl property to infoCourseInstance.json and infoAssessment.json. It’s an error to have both this and allowAccess. Unlike allowAccess which consists of a list of access rules, only a single accessControl object can be used for the assessment as a whole. However, additional accessControl objects can be provided for specific students or sections (groups of students).

UI overview

This is a rough mockup of what the editing UI could look like. This UI will need polishing before it's ready for production. See https://github.com/PrairieLearn/PrairieLearn/pull/10674 for the code.

UI for overrides

Overrides can be specified either for specific students or for sections (groups of students). An override completely replaces the default access control for the people it applies to.

Course instance access control

Example infoCourseInstance.json:

{
    "accessControl": {
        "published": true,
        "startDate": "2024-02-01T00:00",
        "endDate": "2024-05-30T23:59"
    },
    "accessControlOverrides": [
        {
            "sections": ["studentsWithExtendedAccess"],
            "accessControl": {
                "published": true,
                "startDate": "2024-02-01T00:00",
                "endDate": "2024-06-15T23:59"
            }
        }
    ]
}

All properties are optional.

The overall published property is optional, but it defaults to false so that an empty accessControl object means "no access".

Both the startDate and endDate are optional, in which case access there is no limit in that direction. If neither date is specified then the course instance is available forever, which is useful for example courses.

Instead of allowing a list of uids we'll have a separate set of overrides which are apply per-student or per-section (group of students). This is to avoid storing PII (personally identifiable information) in JSON. An override is another entire accessControl object that is scoped to only apply to specific students. If a student belongs to an override then that accessControl is used rather than the default one. Overrides can apply either to a list of students or to a list of sections (groups of students), but not both. Per-section overrides are stored in JSON and synced to the DB, just like the default accessControl, while per-student overrides are only stored in the DB to avoid storing PII in JSON.

Note that there is no longer an institution property. Instead of this, we'll copy PT’s self-enrollment settings, where a course instance will have:

• self_enrollment_allow BOOLEAN - can students self-enroll?
• self_enrollment_key TEXT - random key added to self-enrollment link
• self_enrollment_only_same_institution BOOLEAN - only allow self-enrollments from the same institution

Additionally, we’ll allow instructors to invite student enrollments like in PT, which should not check institutional limitations and thus allows instructors to add students from any institution manually.

Assessment access control

Example infoAssessment.json:

{
    "accessControl": {
        "published": true,
        "dateControl": true,
        "availableDate": "2024-03-14T00:01",
        "dueDate": "2024-03-21T23:59",
        "lateDeadlines": [
            {"date": "2024-03-23T23:59", "credit": 80},
            {"date": "2024-03-30T23:59", "credit": 50}
        ],
        "lateCredit": 30,
        "earlyDeadlines": [
            {"date": "2024-03-17T23:59", "credit": 120},
            {"date": "2024-03-20T23:59", "credit": 110}
        ],
        "zeroCreditPractice": true,
        "durationMinutes": 60,
        "password": "superSecret",
        "hideQuestionsAfterCompletion": true,
        "hideScoreAfterCompletion": true,
        "revealDate": "2024-04-07T00:00",
        "revealScoreDate": "2024-04-01T00:00",
        "prairieTest": [
            {"examUuid": "11e89892-3eff-4d7f-90a2-221372f14e5c"},
            {"examUuid": "896c088c-7468-4045-965b-e8ae134086c2", "readOnly": true}
        ]
    },
    "accessControlOverrides": [
        {
            "sections": ["section1", "section2"],
            "accessControl": {
                "published": false
            }
        }
    ]
}

All properties are optional.

The overall published property is optional, but it defaults to false so that an empty accessControl object means "no access". We might want to also have a separate publishGrades setting (default true) that people can turn on/off when they are manually grading?

The dateControl property determines whether access is controlled by availableDate/dueDate, etc. This property is optional and defaults to true if either availableDate or dueDate is specified, otherwise it defaults to false. If this property is false then it is an error to specify any of availableDate, dueDate, lateDeadlines, lateCredit, earlyDeadlines, durationMinutes, or password. Basically, if dateControl is false then we are only allowing access via PrairieTest. The only reason that dateControl would need to be specified explicitly is to set dateConrol: true and then have both availableDate and dueDate not specified, resulting in an assessment which is always available. See the UI section below for more discussion on dateControl.

We've renamed start/end dates to available/due dates, and there is only a single due date. Both of these are optional, in which case access is controlled by the course instance dates. Maybe releaseDate or publishDate would be better names than availableDate?

The lateDeadlines allow a staggered set of late deadlines for reduced credit, and earlyDeadlines is similar for bonus credit. If lateCredit is set then it allows submissions for this credit after any other deadlines. We could combine late/early deadlines into a single extraDeadlines list but I think the web UI for editing this will be clearer if we separate them and enforce that late deadlines must have credit < 100 and early must have credit > 100.

The zeroCreditPractice setting controls whether students can continue to make submissions even when no credit is available. This defaults to false.

Both durationMinutes and password function just like they do at the moment.

The hideQuestionsAfterCompletion and hideScoreAfterCompletion settings are similar to the current ones, but they apply after "completion" (we may want to disallow the combination of hideQuestionsAfterCompletion: false with hideScoreAfterCompletion: true). An assessment instance is said to be "complete" if we are past the last submission date (and lateCredit is not set) or if the assessment instance is closed. Both of the hidden statuses are revealed after revealDate. After the revealDate the assessment instances are automatically in a read-only mode (the equivalent of the old active:false setting). To reveal just the total score but no question details, use revealScoreDate instead.

The linking with PrairieTest is via a list of examUuids, just like the current setup. This linkage means that access will be additionally granted to students with a corresponding PrairieTest reservation. Properties such as hideQuestionsAfterCompletion continue to apply to PrairieTest exams. The readOnly attribute can be set to restrict an assessment to only be available in a non-editable mode within the corresponding PrairieTest reservation.

There is no mode setting anymore because it's redundant with the PrairieTest UUIDs.

The active setting is not needed anymore. Depending on the use case it's replaced by the published flag, the readOnly flag associated with linked PrairieTest exams, or the hide/reveal settings.

The uids lists are no longer present to avoid storing PII (personally identifiable information) in JSON. These are replaced by the idea of per-student or per-section overrides. An override is another entire accessControl object that is scoped to only apply to specific students. If a student belongs to an override then that accessControl is used rather than the default one. Overrides can apply either to a list of students or to a list of sections (groups of students), but not both. Per-section overrides are stored in JSON and synced to the DB, just like the default accessControl, while per-student overrides are only stored in the DB to avoid storing PII in JSON.

We should add the ability to copy overrides between assessments, or possibly make them apply to assessment sets.

UI for assessments

The web UI for editing an assessment's access control can be divided into three sections:

1. Link with PrairieTest. This section can collapse if there are no linked PrairieTest exams. Expanding it should allow for the addition of exam_uuid links.
2. Allow access by date. The collapse of this section is determined by the boolean date_control. If date_control is true then we can expand the section to set available_date, due_date, late_credit_perc, early deadlines, late deadlines, duration_minutes, and password. Otherwise the section collapses and these are all NULL or deleted.
3. Remaining settings, which are always visible, including hideQuestionsAfterCompletion, hideScoreAfterCompletion, revealDate, and revealScoreDate.

We should also have a status message at the top to show whether the assessment is published or not, with a button to toggle this.

Everywhere we list assessments, it would be great to have an icon next to each of them to indicate whether it is published. If an assessment is not published, then every page about it should have a banner saying "This is not published to students" with a button or link to change that.

UI for students

On student assessments page we should list assessments with their due dates. If they have early or late deadlines then we should put an icon next to the due date with a popover to list the extra dates. Similarly, if the student has overrides then they should see an icon with a popover to list the overrides.

DB schema for course instances

Create a new table course_instance_access_controls and add a foreign key from course_instances to point to this new table. Require these keys to be UNIQUE but allow them to be NULL. If an assessment has a non-NULL access control key then we assume it's using the new access control scheme. Otherwise fall back to using the legacy access rules.

Also add new tables for overrides and a join table for the case when an override applies to multiple students.

course_instances:
    course_instance_access_control_id BIGINT UNIQUE REFERENCES course_instance_access_controls ON DELETE SET NULL ON UPDATE CASCADE

course_instance_access_controls:
    published BOOLEAN NOT NULL DEFAULT FALSE
    start_date TIMESTAMP WITH TIME ZONE
    end_date TIMESTAMP WITH TIME ZONE

enum course_instance_override_type:
    user
    section

course_instance_overrides:
    course_instance_id BIGINT NOT NULL REFERENCES course_instances ON DELETE CASCADE ON UPDATE CASCADE
    access_control_id BIGINT NOT NULL REFERENCES course_instance_access_controls ON DELETE CASCADE ON UPDATE CASCADE
    type course_instance_override_type NOT NULL
    order INTEGER NOT NULL

course_instance_override_users:
    course_instance_override_id BIGINT NOT NULL REFERENCES course_instance_overrides ON DELETE CASCADE ON UPDATE CASCADE
    user_id BIGINT NOT NULL REFERENCES users ON DELETE CASCADE ON UPDATE CASCADE

course_instance_override_sections:
    course_instance_override_id BIGINT NOT NULL REFERENCES course_instance_overrides ON DELETE CASCADE ON UPDATE CASCADE
    section_id BIGINT NOT NULL REFERENCES sections ON DELETE CASCADE ON UPDATE CASCADE

DB schema for assessments

Create a new table assessment_access_controls and add a foreign key from assessments to point to this new table. Require these keys to be UNIQUE but allow them to be NULL. If an assessment has a non-NULL access control key then we assume it's using the new access control scheme. Otherwise fall back to using the legacy access rules.

Also add new tables for overrides and a join table for the case when an override applies to multiple students.

assessments:
    assessment_access_control_id BIGINT UNIQUE REFERENCES assessment_access_controls ON DELETE SET NULL ON UPDATE CASCADE

assessment_access_controls:
    date_control BOOLEAN NOT NULL
    available_date TIMESTAMP WITH TIME ZONE
    due_date TIMESTAMP WITH TIME ZONE
    late_credit_perc INTEGER CHECK (late_credit_perc > 0 AND late_credit_perc < 100)
    zero_credit_practice BOOLEAN NOT NULL DEFAULT FALSE
    duration_minutes INTEGER CHECK (duration_minutes > 0)
    password TEXT
    CONSTRAINT assessment_date_control_force_null CHECK (
        CASE WHEN NOT date_control THEN (available_date IS NULL AND due_date IS NULL AND late_credit_perd IS NULL AND time_limit_minutes IS NULL AND password IS NULL)
        ELSE TRUE
    )
    hide_questions_after_completion BOOLEAN NOT NULL DEFAULT FALSE
    hide_score_after_completion BOOLEAN NOT NULL DEFAULT FALSE
    reveal_date TIMESTAMP WITH TIME ZONE
    reveal_score_date TIMESTAMP WITH TIME ZONE

assessment_late_deadlines:
    assessment_access_control_id BIGINT NOT NULL REFERENCES assessment_access_controls ON DELETE CASCADE ON UPDATE CASCADE
    date TIMESTAMP WITH TIME ZONE NOT NULL
    credit_perc: INTEGER NOT NULL CHECK (credit_perc > 100)

assessment_early_deadlines:
    assessment_access_control_id BIGINT NOT NULL REFERENCES assessment_access_controls ON DELETE CASCADE ON UPDATE CASCADE
    date TIMESTAMP WITH TIME ZONE NOT NULL
    credit_perc: INTEGER NOT NULL CHECK (credit_perc > 0 AND credit_perc < 100)

assessment_exam_uuids:
    assessment_access_control_id BIGINT NOT NULL REFERENCES assessment_access_controls ON DELETE CASCADE ON UPDATE CASCADE
    exam_uuid UUID NOT NULL
    read_only BOOLEAN NOT NULL DEFAULT FALSE

enum assessment_override_type:
    user
    section

assessment_overrides:
    assessment_id BIGINT NOT NULL REFERENCES assessments ON DELETE CASCADE ON UPDATE CASCADE
    assessment_access_control_id BIGINT NOT NULL REFERENCES assessment_access_controls ON DELETE CASCADE ON UPDATE CASCADE
    type assessment_override_type NOT NULL
    order INTEGER NOT NULL
    UNIQUE (assessment_id, type, order)

assessment_override_users:
    assessment_override_id BIGINT NOT NULL REFERENCES assessment_overrides ON DELETE CASCADE ON UPDATE CASCADE
    user_id BIGINT NOT NULL REFERENCES users ON DELETE CASCADE ON UPDATE CASCADE

assessment_override_sections:
    assessment_override_id BIGINT NOT NULL REFERENCES assessment_overrides ON DELETE CASCADE ON UPDATE CASCADE
    section_id BIGINT REFERENCES sections ON DELETE CASCADE ON UPDATE CASCADE

sections:
    course_instance_id BIGINT NOT NULL REFERENCES course_instances ON DELETE CASCADE ON UPDATE CASCADE
    name TEXT NOT NULL
    description TEXT
    UNIQUE (course_instance_id, name)

section_users:
    section_id BIGINT NOT NULL REFERENCES sections ON DELETE CASCADE ON UPDATE CASCADE
    user_id BIGINT NOT NULL REFERENCES users ON DELETE CASCADE ON UPDATE CASCADE
    UNIQUE (section_id, user_id)

Access control logic for course instances

Check whether this course instance has a non-null access_control_id. If it does, then use that access control settings. Otherwise, fall back to the legacy allowAccess rules.

To determine which access control settings apply to a given student we choose:

1. If there is a per-student override for this student, then use that. If there are multiple, use the one with the lowest value of order.
2. Similarly, but for per-section overrides.
3. Otherwise, use the default access control settings for the assessment.

The published flag overrides all other access control. If it is false then we immediately block access, otherwise proceed.

If the current date is between startDate and endDate (or one or the other is NULL) then grant access, otherwise not.

Access control logic for assessments

First check whether we have access to the course instance. If we do, proceed below.

Check whether this assessment has a non-null access_control_id. If it does, then use that access control settings. Otherwise, fall back to the legacy allowAccess rules.

To determine which access control settings apply to a given student we choose:

1. If there is a per-student override for this student, then use that. If there are multiple, use the one with the lowest value of order.
2. Similarly, but for per-section overrides.
3. Otherwise, use the default access control settings for the assessment.

The published flag overrides all other access control. If it is false then no aspect of the assessment is visible to the student. Assuming published: true, we proceed below.

PrairieTest active: We define that an assessment has prairietest_active if:

• We have a linked PrairieTest exam
• AND that exam has a reservation for the current student
• AND the current time is between the reservation start_access/end_access times

Available: We define that an assessment is available if:

• prairietest_active
• OR date_control is true AND:
  • available_date is NULL
  • OR now() > available_date
• OR we have an active override

Positive credit: We define that we have positive_credit if:

• The assessment is available AND:
  • We have prairietest_active
  • OR date_control is true AND:
    • due_date is NULL
    • OR now() is before the maximum of the due date and all late deadlines
    • OR lateCredit is set
  • OR we have an active override with credit_perc > 0

Creation: Assessment instances can be created whenever we have positive_credit.

Editing: An assessment instance can be edited if:

• We have positive_credit AND the assessment instance is open
• OR it is available AND zero_credit_practice is true

Completion: We say that an assessment instance is "complete" if:

• it is not open
• OR positive_credit is false

Viewing:

• If hideQuestionsAfterCompletion is false (the default) then an assessment instance can be viewed if it is available
• If hideQuestionsAfterCompletion is true then an assessment instance can be viewed if:
  • it is available
  • AND:
    • The assessment instance is editable
    • OR the current time is after revealDate

Score viewing:

• If hideScoreAfterCompletion is false (the default) then we can always view the score.
• If hideScoreAfterCompletion is true then we can view the score if:
  • The assessment instance is open
  • OR the current time is after revealDate
  • OR the current time is after revealScoreDate

Migration plan

A course instance or assessment can have either allowAccess or accessControl, but not both. Initially we'll support both options on an equal basis.

In the web UI I'm thinking we could add a "convert to new access control settings" button on the assessment access page. This would use some heuristics to guess at the appropriate values to fill in for the various fields and then drop the user into an editing page with the guessed values filled in. The user can then fix up any mistakes and save the result. I don't think this auto-converter needs to be very complete. It would be enough to pull out the main dates (availableDate, dueDate), just to have a starting point.

At some future date we should start warning that the old allowAccess rules are deprecated. This warning can be done on sync for any assessment which has an access rule with a date in the future. At some even later date we can switch to ignoring allowAccess completely and remove the corresponding code. That is, we'll still allow the allowAccess property to be set in an assessment or course instance, but it's presence will have no effect. It won't be an error to not have a new-style accessControl, but that assessment will just not be accessible. This will be the standard model for old assessments in past course instances.

Example use case 1: Homework

This is a homework which is released on a given date, is due a week later, and offers both bonus credit for doing it early and reduced credit for doing it late. After the final late deadline, questions can still be practiced for zero points.

{
    "accessControl": {
        "published": true,
        "availableDate": "2024-03-14T00:01",
        "dueDate": "2024-03-21T23:59",
        "earlyDeadlines": [
            {"date": "2024-03-17T23:59", "credit": 120},
            {"date": "2024-03-20T23:59", "credit": 110}
        ],
        "lateDeadlines": [
            {"date": "2024-03-23T23:59", "credit": 80},
            {"date": "2024-03-30T23:59", "credit": 50}
        ],
        "zeroCreditPractice": true
    }
}

Example use case 2: Limited-duration exam

This is an exam where the student can start it anytime between the available/due dates and then has a fixed time to complete the exam. Once the exam is over, the assessment instance is closed, which prevents further editing, but the exam is still available to view indefinitely.

{
    "accessControl": {
        "published": true,
        "availableDate": "2024-03-14T00:01",
        "dueDate": "2024-03-21T23:59",
        "durationMinutes": 60
    }
}

To hide the score and/or questions after the exam is over, but reveal them on a later date, extra properties can be used like:

    {
        "hideQuestionsAfterCompletion": true,
        "hideScoreAfterCompletion": true,
        "revealDate": "2024-04-01T00:00"
    }

Example use case 3: PrairieTest exam

This is an exam where access is controlled by PrairieTest. Once a student finishes the exam they can see their score and the questions while their PrairieTest reservation is open. After their reservation is over they can see their total score but not the assessment details.

{
    "accessControl": {
        "published": true,
        "prairieTestExamUuids": ["11e89892-3eff-4d7f-90a2-221372f14e5c"]
    }
}

Common customizations:

• To allow students to view their exams outside of PrairieTest after a given date, we can additionally set revealDate.
• To block students from viewing their scores after the exam, set "hideScoreAfterCompletion": true.
• To block students from viewing the questions after completion, even within the PrairieTest reservation, set "hideQuestionsAfterCompletion": true.

Example use case 4: PrairieTest exam with some students taking it outside of PrairieTest

We want to run an exam via PrairieTest but we have a group of students who are taking the exam in a disability center without PrairieTest proctoring.

• Set up the accessControl for a PrairieTest exam.
• Use overrides to give access to the students who are taking the exam in the disability center.

Example use case 5: In-class exam with a few students taking it via PrairieTest

We want to run an in-class exam, but students with disability accommodations will be sent to a PrairieTest testing center.

• Set up the accessControl for an in-class exam and also link it to PrairieTest.
• In PrairieTest make the exam non-self-reservable, but give overrides to the students with accommodations.

Example use case 5: PrairieTest exam with review sessions in PrairieTest

We want students to be able to return to the CBTF for a second session where they can view (but not start or edit) their assessment instances.

• Set prairieTestExamUuids to both the PrairieTest exam and the review exam.
• In PrairieTest, set the review exam to be ReadOnly.

Example use case 6: Extended time overrides for a group of students

We have a group of students who should all receive 50% extra time for an exam, and we are not using PrairieTest to manage this.

• Set up the exam as usual for regular students.
• Add overrides for the students with extra time, or put them in a section and make an override for that section.

Example use case 7: Homework for providing a cheat sheet in the CBTF

Students should be able to edit questions outside of the CBTF, especially to upload a PDF file or similar, and then access a read-only version in a CBTF exam.

• Set up the exam with two different ways of access: an availableDate/dueDate for when the cheat sheets can be uploaded, and a PrairieTest examUuid for the exam with readOnly set to `true.
• If course staff want to check the cheat sheets before the exam then the dueDate will need to be set early enough that they can go through the submissions, perhaps using manual grading to coordinate this.

Example use case 8: Exam questions revealed and then hidden again

From @ffund: https://github.com/PrairieLearn/PrairieLearn/issues/10318#issuecomment-2327191101

After an exam, Fraida reveals the exam questions to students for a limited period of time and then hides them again. This encourages students to look at the exam immediately, while the content is still fresh and relevant and they might learn something from the feedback.

At the moment this is not supported by the new access control scheme. It can always be implemented manually by the instructor going to the assessment and turning off "reveal questions" after the allotted time, but it can't be pre-programed ahead of time.

Example use case 9: Different passwords for different exam sessions

From @ffund: https://github.com/PrairieLearn/PrairieLearn/issues/10318#issuecomment-2327191101

Fraida has multiple exam sessions and students are free to show up to any of them. Each session has a different password set and when the student arrives they are told the appropriate password by the TA for that session.

This can't be directly supported by the new access control scheme, because overrides are based on student identity and not time period. However, a substitution system might be to make an empty override for each exam session, and when students show up the TA can add them to the override for that session. This is more secure than the password system, although it is also somewhat more work for the TA.

Another alternative is to use PrairieTest for the exam sessions, especially with the new "forced reservations" capability that allows proctors to add students to sessions at any time (coming soon).

Example use case 10: Non-real-time-grading exams without a time limit

Some UBC instructors have quizzes where students have a fixed time period to complete the quiz (start/end dates) but the quiz is non-real-time-graded. Grades are then revealed after the end date. The proposed system accommodates this usage, but we should check that the autoclose code works correctly after the due date.

Similarly, for lab sections they have the same setup, but with different due dates per section. This should also work fine.

[jonatanschroeder]

We might want to also have a separate publishGrades setting (default true) that people can turn on/off when they are manually grading?

Isn't that what revealDate is supposed to handle?

        "lateCredit": 30,

If this means what I'm assuming, I'd suggest replacing this with an entry in lateDeadlines that has no date.

For overrides, are you considering "groups" that are not assessment groups (e.g., a group that comprises "students with 1.50 exam time")? If that is the case, it would be good to allow the exam time limit to be expressed as a percentage in the override. Also, this needs to have an easy way to copy from one assessment to the next (or to apply to a set of assessments, e.g., all assessments in the set "Quizzes"), so that when I create a new assessment locally then sync in production I don't need an additional step that I may forget to run.

[mwest1066]

We might want to also have a separate publishGrades setting (default true) that people can turn on/off when they are manually grading?

Isn't that what revealDate is supposed to handle?

I guess my main question here is whether we can/should try and separate hiding scores from hiding everything about the assessment from students? How feasible is it to show students the questions and their submissions, but not scoring information?

        "lateCredit": 30,

If this means what I'm assuming, I'd suggest replacing this with an entry in lateDeadlines that has no date.

I kind of like this, but we'd need to add extra validation logic to make sure that at most one of the entries has no date, and in the UI I think it would be better to separate out lateCredit. For example, I'm thinking of something like:

Late credit:

• Until Apr 10: 80% credit
• Until Apr 14: 50% credit
• [Add another late deadline]
• Late credit after all deadlines: 30%

For overrides, are you considering "groups" that are not assessment groups (e.g., a group that comprises "students with 1.50 exam time")? If that is the case, it would be good to allow the exam time limit to be expressed as a percentage in the override.

Good point, added.

Also, this needs to have an easy way to copy from one assessment to the next (or to apply to a set of assessments, e.g., all assessments in the set "Quizzes"), so that when I create a new assessment locally then sync in production I don't need an additional step that I may forget to run.

Yep, added.

[nwalters512]

Overall this seems reasonable, certainly much simpler for instructors to understand! My concerns about taking away some amount of control from power users stands, but I'm hopeful that early feedback + work on our part to prove this out (see below) will ease those concerns.

In the web UI I'm thinking we could add a "convert to new access control settings" button on the assessment access page. This would use some heuristics to guess at the appropriate values to fill in for the various fields and then drop the user into an editing page with the guessed values filled in.

I like this as an idea. Depending on how robustly we build this thing, this could also be a good test for ourselves to prove that we can handle the vast majority of access rule scenarios. That is, we could run this against all sets of access rules in production and figure out if we're missing any common cases. This would of course involve a fair amount of manual work, but might be useful to us (assuming that the vast majority of users won't review this proposal).

We'll need to add a "review" setting in PT exams to control whether the access granted is read/write or just read.

This makes enough sense to me. I was briefly wondering if this should instead be part of the PL access control block instead, but given that instructors are already hopefully used to configuring the assessment duration in PT, it hopefully makes sense to put this new access control option there. We'd want to make sure that this can at least in theory play nicely with third-party LMSes via PT webhooks.

DurationPerc (only valid if the base config has a duration) OR durationPerc (percentage to scale the base config duration)

These seem to be the same property + description, what am I missing?

Separately, we should standardize on either duration or timeLimit. I'm inclined to pick the former since that's what PT uses.

If we have an active PrairieTest reservation for a linked UUID (checked in, started, between start/end times) then we also say that we have "positive credit".

Does this imply 100% credit for any PT-linked assessment, or would we still compute credit based on earlyDeadlines/lateDeadlines/lateCredit?

[nwalters512]

One use case to consider: making a "homework" available for view, but not submit, in a PT exam. The concrete use case is that some folks are using a separate assessment as a means for students to upload a "cheat sheet" that they can then access in the CBTF. Currently, this would be achieved by adding two access rules to that homework: one that lets them submit until e.g. a day before the exam window starts, and another with a PT UUID and "active": false. I'm not sure how we'd achieve that with this new system. I guess they could do it if they don't care about it being inactive in PT?

[ffund]

I have some feedback from the POV of a person who is not a PL developer, just a regular user :sweat_smile: I find this wayyyyyyyy more confusing than the old way. Maybe it's just me!

[mwest1066]

We'll need to add a "review" setting in PT exams to control whether the access granted is read/write or just read.

This makes enough sense to me. I was briefly wondering if this should instead be part of the PL access control block instead, but given that instructors are already hopefully used to configuring the assessment duration in PT, it hopefully makes sense to put this new access control option there. We'd want to make sure that this can at least in theory play nicely with third-party LMSes via PT webhooks.

In the updates just now I changed to putting the "read-only" setting in the PL access control block. This is necessary to support the cheat-sheet use case, where we want to have two different PL assessments (the cheat sheet HW and the actual exam) available in the same PT exam, but one of them should be read-only and the other not.

DurationPerc (only valid if the base config has a duration) OR durationPerc (percentage to scale the base config duration)

These seem to be the same property + description, what am I missing?

To simplify things I just replaced this with an exact copy of the property that's being overridden (time_limit_minutes). This replicates the current functionality where instructors always need to specify the exact number of minutes for each group of students, rather than having a base duration and percentage changes (as PT uses). I don't have any particular objection to supporting percentage changes at some point, but it adds complexity and doesn't seem essential at the moment.

Separately, we should standardize on either duration or timeLimit. I'm inclined to pick the former since that's what PT uses.

I could get behind that. I'll change it now.

If we have an active PrairieTest reservation for a linked UUID (checked in, started, between start/end times) then we also say that we have "positive credit".

Does this imply 100% credit for any PT-linked assessment, or would we still compute credit based on earlyDeadlines/lateDeadlines/lateCredit?

It implies 100% credit for any PT-linked assessment. I haven't observed a desire from instructors to give reduced credit for late PT exams. If we did want to support that, I actually think it would be better to do it from within PT (where overrides are being given) rather than needing to both give an override in PT and also add date ranges in PL. If we did support reduced-credit overrides in PT, I think that instructors would want to mix some overrides at 100% (e.g., student was sick) and some at reduced credit (e.g., student missed the exam so they get an 80%-credit override). These different overrides would overlap in time, so using dates to try and assign credit would be unnatural.

One use case to consider: making a "homework" available for view, but not submit, in a PT exam. The concrete use case is that some folks are using a separate assessment as a means for students to upload a "cheat sheet" that they can then access in the CBTF. Currently, this would be achieved by adding two access rules to that homework: one that lets them submit until e.g. a day before the exam window starts, and another with a PT UUID and "active": false. I'm not sure how we'd achieve that with this new system. I guess they could do it if they don't care about it being inactive in PT?

Great point! I added this as example use case 7. I also added the dateControl and readOnly properties and adjusted some of the logic to account for this case.

[mwest1066]

I have some feedback from the POV of a person who is not a PL developer, just a regular user 😅 I find this wayyyyyyyy more confusing than the old way. Maybe it's just me!

Thanks for the feedback, @ffund! I really value your input here, but would you mind expanding on this a bit to help me understand? In particular, could you explain one or more of your use cases, how you achieve them at the moment, and what you think is more confusing about this proposed new systems for your use cases? As much detail as possible would be super helpful. Thanks!!

[ffund]

1. I think this is the main thing that feels much more complicated to me -

Generally speaking, in the "old" approach each rule had a data range and then specific options associated with it, so I could see/understand exactly what options apply in each date range.

In the "new" approach, there are a bunch of dates and a bunch of options, and I don't fully understand which options apply at which times. e.g., in the example, do students need the password to review their work after the revealDate? Is there still a durationMinutes during the zeroCreditPracticeTime? Can I have an assessment with a 100% credit time interval and scores hidden, and then a 0% credit time interval with scores visible? I don't know which options can/do apply during which date ranges.

Presumably this will be documented somewhere, but I find it difficult to understand already, and I can imagine it will get even complicated as more options are inevitably added.

(zeroCreditPracticeTime feels like it should be a lateDeadline with 0 credit, not an option, BTW. And I agree with Jonatan about the lateCredit thing also being a lateDeadline.)

Also, some personal use cases that I don't see how to translate to this system -

• use case 2 (limited-duration exam) but with the "review" period closing after some time. e.g. students have one week to review their exam after the revealDate (original exam may be in PT or not, review is outside of PT) -

    "allowAccess": [
            {
                "mode": "Exam",
                "examUuid": "a19d4358-0d7e-4eb2-8be1-464297a50379",
                "credit": 100
            },
            {
                "mode": "Public",
                "startDate": "2024-03-08 09:00:00",
                "endDate": "2024-03-15 23:59:59",
                "active": false
        }

• multiple time slots for the same exam, with different passwords. e.g.

    "allowAccess": [
        {
            "startDate": "2023-12-14T09:50:00",
            "endDate": "2023-12-14T11:10:00",
            "credit": 100,
            "timeLimitMin": 15,
            "password": "XXX"
        },
        {
            "startDate": "2023-12-14T12:00:00",
            "endDate": "2023-12-14T13:10:00",
            "credit": 100,
            "timeLimitMin": 15,
            "password": "YYY"
        }

Re the question of "restrict this assessment to a specific set of students" - FYI this is something I used during "makeup" assessments in an alt grading scenario.

[mwest1066]

@ffund thanks for the detailed feedback, this is great!

Generally speaking, in the "old" approach each rule had a data range and then specific options associated with it, so I could see/understand exactly what options apply in each date range.

I very much appreciate this point. The general intention with the proposed new system is that there is one set of options that apply generally (and I appreciate your point that documentation is needed to explain exactly what applies when), and then we use overrides to handle special cases.

I should also note that part of our motivation in rethinking access control is that the old rules are in fact not so obviously orthogonal to each other in all cases. For example, hiding and revealing scores, allowing zero-credit practice, etc, all have non-obvious interactions between multiple access rules, which often confuse users (and us!).

However, I'll think more about whether we can solve these current issues with a system like the current access rules.

Presumably this will be documented somewhere, but I find it difficult to understand already, and I can imagine it will get even complicated as more options are inevitably added.

I can appreciate this viewpoint. I should mention that our intention here is that most users will configure this new access control via a web UI, which will have inline explanations for what each option does and when it applies. This is part of the intention in encoding more semantic meaning in the access control itself, to guide users in the UI. We are trying to move away from the need for people to read the documentation when writing JSON by hand (that will always remain an option, of course, but we've found that most instructors don't want to do this).

• use case 2 (limited-duration exam) but with the "review" period closing after some time. e.g. students have one week to review their exam after the revealDate (original exam may be in PT or not, review is outside of PT) -

Can I ask why you remove access again after one week? Is this to avoid students coming to complain near the end of term?

• multiple time slots for the same exam, with different passwords. e.g.

Is this for different class sections? Do you additionally restrict to particular users, or just by time? I'm thinking we could handle this case with an override that applies to all students.

Re the question of "restrict this assessment to a specific set of students" - FYI this is something I used during "makeup" assessments in an alt grading scenario.

Makes sense, thanks for the example. This can already be handled with overrides in the proposed new system, but the UI might be clearer if we allow user-restrictions on the access control itself.

[ffund]

• use case 2 (limited-duration exam) but with the "review" period closing after some time. e.g. students have one week to review their exam after the revealDate (original exam may be in PT or not, review is outside of PT) -

Can I ask why you remove access again after one week? Is this to avoid students coming to complain near the end of term?

To encourage students to look at it immediately, while the content is still fresh and relevant and they might learn something from the feedback. I do this for all exams.

(And also yes, to reduce grade grubbing at the end of the semester.)

• multiple time slots for the same exam, with different passwords. e.g.

Is this for different class sections? Do you additionally restrict to particular users, or just by time? I'm thinking we could handle this case with an override that applies to all students.

It could be different sections. But what I had in mind was "to take the quiz, show up to the beginning of any of the course staff's office hours". Any student can go to any of the times, access control is via the TA giving out the time-specific password.

[mwest1066]

Can I ask why you remove access again after one week? Is this to avoid students coming to complain near the end of term?

To encourage students to look at it immediately, while the content is still fresh and relevant and they might learn something from the feedback. I do this for all exams.

(And also yes, to reduce grade grubbing at the end of the semester.)

Makes sense, I can appreciate this. Let me think about how to support this.

• multiple time slots for the same exam, with different passwords. e.g.

Is this for different class sections? Do you additionally restrict to particular users, or just by time? I'm thinking we could handle this case with an override that applies to all students.

It could be different sections. But what I had in mind was "to take the quiz, show up to the beginning of any of the course staff's office hours". Any student can go to any of the times, access control is via the TA giving out the time-specific password.

Makes sense. We definitely want to continue to support this.

[mwest1066]

It could be different sections. But what I had in mind was "to take the quiz, show up to the beginning of any of the course staff's office hours". Any student can go to any of the times, access control is via the TA giving out the time-specific password.

Following up on this further, is there a reason that you aren't using PrairieTest for this? The time-specific passwords are a bit insecure (a student can message a friend to tell them the password) and managing lots of exam sessions like this is well-handled by PrairieTest, I think?

Also, how do you handle extended-time accommodations with this setup? This is another aspect that I think PrairieTest would make much easier.

[ffund]

PT is too much overhead for these quick, low stakes quizzes. (Especially without a CBTF.)

For students with extended time, I'd have a separate (earlier in the file) rule with the student's ID, with a time range that includes all of the possible times (so they can show up whenever). Let's say students X, Y, Z show up to an office hour. X has extended time. The TA can announce the "general" quiz password for that time, and separately tell X their "personal" password.

At the scale we were doing this at, it worked (students weren't trying to share the passwords, and the TAs would be able to tell if they were). I can see how at a larger scale, it wouldn't work.

[mwest1066]

At the scale we were doing this at, it worked (students weren't trying to share the passwords, and the TAs would be able to tell if they were). I can see how at a larger scale, it wouldn't work.

Thanks, that's helpful context.

[mwest1066]

@nwalters512 and I discussed this and I've updated the issue description to reflect our conclusions. The main change is that overrides can now override any property of the default access control. This means that you can give completely custom access to either specific students or to sections (groups) of students.

I also added some screenshots of an editing mockup. Hopefully this helps to get a sense of which options are available and what this might look like.

[mwest1066]

@ffund: I added your two use cases above as Example use cases 8 and 9 in the issue description. As you can see there, the current plan would require instructor action for the limited-release scenario (case 8), but there is a potential workaround for the multiple exam sessions scenario (case 9). I'd really appreciate your feedback on these (and anything else, of course)!

[lslavice]

I think that what you're proposing has a lot of thought gone into it and I am confident that we can adapt to the new system. I am 100% okay relying on the expertise of you all who have been working hard on this!! And it would bring PL in line with how instructors are accustomed to assigning deadlines/control in other platforms (here is the general case, now assign overrides for specific groups/individuals). Two things to relay:

1. Please please do not remove the ability to configure access via JSON files. We have 60 PL assignments and upwards of 800 students each term in the one course I teach. One of the biggest strengths of PL has been NOT having to click a million times to copy a course to the next term and update deadlines (looking at you, Canvas). We're totally fine with using a differently formatted JSON file and to set up student groups for overrides (like students needing extra time on assessments) and all that. But I will seriously break down and cry if we have to go back to clicking things on a website and it taking hours and hours instead of like 10 minutes. Also, being able to edit everything in JSON files and then preview it locally to check and fix mistakes is wonderful.

2. Please have a way to verify individual students for overrides beyond just their name. And yes, this comes from experience lol. One year, we had two students with the EXACT SAME first and last name. We had to be very very careful to always refer to them by their "uniqname" (the part of their email before the @umich.edu). And even doing this we STILL managed to accidentally mixup their mid-term exams, giving each one the other's graded exam, which is clearly a terrible thing. With the emails in the JSON files, we don't have this problem, but I totally understand not wanting to have PII in the individual JSON files. I'm not sure how you're imagining setting up the student groups (e.g. extended time) or selecting an individual student for an override (dropdown menu?), but somehow I'd like to be able to verify via two different data points that I have the correct student (e.g. name and email address). The "exact same name" students were the worst with this, but it's even confusing trying to keep all the students with the same/similar first names straight. We have 11 students named Sophia/Sophie/Sofia this semester!! We default to using their uniqnames (i.e. emails) to keep everyone straight.

And then one recommendation for clarity: I would add a % sign after the number for the credit for each of the deadline options. So, "80%" instead of 80, or "100%" instead of 100. To make it clear that it is a percentage and not points. If you have a good reason to not have the % displayed, then that is okay, too. We can adapt.

[lslavice]

I apologize if this is explained up above and I'm just not finding it...

Can you explain (or show a mockup) of what students will see if an assessment is published but is not yet released via it's release date?

The reason I am asking is that I really like that we can get everything set up before the semester (aka "it's all published and ready to go") and then PL "reveals" assignments/assessments to students according to the release date. And we don't have to go and click a box that says "publish" at like 12:01am. But if under the new system all our assignments that are published (which would be all of them) all show up for students from the beginning of the semester, but then they get access errors if they click on something that isn't released yet, that will be annoying, confusing, and spark so many emails and Piazza posts lolol.

[mwest1066]

@lslavice thanks for the detailed feedback! This is super helpful.

1. Please please do not remove the ability to configure access via JSON files.

Don't worry! We strongly believe in having JSON files in git repos as the basic data structure for PL. The addition of the web UI will always be just an interface to the underlying JSON. You can see an example of the JSON for the new access control above, in the "Assessment access control" section below the screenshots.

2. Please have a way to verify individual students for overrides beyond just their name.

Excellent point, and I appreciate your example to remind us of this. We will continue to make sure to always list both UIDs (like [email protected]) and full names (Matthew West).

I would add a % sign after the number for the credit for each of the deadline options.

Absolutely!

Can you explain (or show a mockup) of what students will see if an assessment is published but is not yet released via it's release date?

This is a really good point and I think it's an oversight in the proposed scheme. At the moment we don't have a way to say "show students that there is an upcoming (unreleased) assessment and tell them the date it will be released". I'm thinking that there are three stages that an assessment could go through:

1. Students can't see that the assessment even exists. It doesn't appear for them in any way.
2. Students can see that the assessment exists, but they can't yet click on it and it has a note saying something like "will be available starting June 4th".
3. The assessment is actually available for students to answer questions.

At the moment, we intended that assessments would go straight from Stage 1 to Stage 3. But I know that many instructors also want to have their assessments in Stage 2.

We could enable this in one of two ways:

1. Have a boolean option called something like visibleBeforeAvailable (default false). This would make an assessment be in Stage 2 whenever it is published.
2. Have a date/time setting called visibleDate. When set, this would transition the assessment to Stage 2 on this date. This has the advantage that you can delay the Stage 1->2 transition until a specific date, but I suspect that this is not really a thing that we want to do.

@lslavice do you have any preference here? Is this Stage 1/2/3 distinction what you were getting at with your question?

[nwalters512]

The active setting is not needed anymore. Depending on the use case it's replaced by the published flag, the review flag inside PrairieTest, or the hide/reveal settings.

I think the readOnly flag has replaced the idea that PT would have a review flag; if that's the case, update this sentence to avoid confusion.

[mwest1066]

I think the readOnly flag has replaced the idea that PT would have a review flag; if that's the case, update this sentence to avoid confusion.

Good catch! Done.

[lslavice]

@mwest1066

Re: JSON. Thank you thank you! hahaha. I did see the JSON example and figured that the intent was to keep that method of interacting with PL, but I wanted to make sure that it definitely had support from at least one of your devoted fans! 🙂

Re: When/how assessments show up. I am totally fine with going from Stage 1 (doesn't show up at all) to Stage 3 (it's there, go do it) like we have now. In many ways, I like this best for students because their situation is "do you see this on PrairieLearn? Then it's open, go do it!!". And they can quickly scan down their list of assessments and make sure they are all up to date. I have all the assignments (both open and not) listed in both Canvas and on our course website, so I don't strongly feel the need to also have unavailable assignments shown in PL.

I am a big big fan of giving instructors flexibility, though. If people find it helpful to have an assignment go from Stage 1 --> 2 --> 3, I'd like them to be able to do that. If they want to just go from Stage 1 --> 3, let's have a way to do that. And if they want to have "delay visibility" in Stage 2 (like you describe above), I can see a use case for that too. For example, we have practice assessments that are available a week before the "real" assessment. I could see us showing the real assessment in PL as "upcoming, but not yet open" as a reminder to students that it's on the horizon and not to forget about it. Others may have similar situations.

Oh, and yes, your description of Stage 1/2/3 is what I was asking about!

[nwalters512]

There's a specific use case we should make sure we have a good story for with this new system: disabled real-time grading in a CBTF. Here's the scenario:

An instructor has disabled real-time grading and opted to now allow students to see their scores after they finish an assessment:

        {
            "mode": "Exam",
            "examUuid": "eca1acfd-3f98-412a-9b0b-750435dcaea6",
            "credit": 100,
            "showClosedAssessment": false,
            "showClosedAssessmentScore": false
        },

The problem arises when a student in the CBTF doesn't grade and close their assessment before their CBTF access expires. When that happens, they end up in a weird state. The assessment instance is still open (because they didn't close it in time), and they can't close it (because they no longer have access). Until the 6-hour auto-close timer elapses, the showClosedAssessment... settings won't apply, because it's not closed, so the default 0/X score will show on the gradebook and on the assessment list. This is understandably confusing to students, which in turn puts a support burden on instructors, who have to respond to panicked emails.

We should make sure that the revealDate and revealScoreDate settings play nicely with disabled real-time grading, or at least that they can be used to avoid this problem.

[reteps]

/*
Certain keys should override the entire object, and not merge with the parent object. Other keys
should merge with the parent object. TODO: flesh out database schema, and especially
what the rules are for overrides.
*/

const accessControl = {
  // These three keys can't/shouldn't be inherited by overrides/other rules
  targets: ['sectionA'],
  enabled: true, /* should you consider this access rule? */ // default true if not set
  blockAccess: false, /* short circuit, deny access if applies */ // default false if not set

  // All other keys are inherited by overrides/other rules

  listBeforeRelease: true, /* student can: can see the title, click into the assessment */
  /* Alternative name may be e.g. dateVisibleBeforeRelease */


  dateControl: {
    enabled: true,
    releaseDateEnabled: true,
    releaseDate: '2024-03-14T00:01',

    dueDateEnabled: true,
    dueDate: '2024-03-21T23:59',
    
    earlyDeadlinesEnabled: true,
    earlyDeadlines: [
      { date: '2024-03-17T23:59', credit: 120 },
      { date: '2024-03-20T23:59', credit: 110 },
    ],

    lateDeadlinesEnabled: true,
    lateDeadlines: [
      { date: '2024-03-23T23:59', credit: 80 },
      { date: '2024-03-30T23:59', credit: 50 },
    ],

    /* If allowSubmissions is true, the afterComplete section will never apply */
    afterLastDeadline: {
      /* If allowSubmissions is false, it doesn't make sense for creditEnabled or credit to be set */
      allowSubmissions: true,

      /* creditEnabled is a UI-only boolean. It doesn't participate in logic. */
      creditEnabled: true,
      credit: 30,
   },

    durationMinutesEnabled: true,
    durationMinutes: 60,
    passwordEnabled: true,
    /* If the passwordEnabled field is missing, then if password is set,
        passwordEnabled is true, otherwise it is inherited. */
    password: 'superSecret',

  },
  prairieTestControl: {
    enabled: true,
    exams: [
      { examUuid: '11e89892-3eff-4d7f-90a2-221372f14e5c' },
      { examUuid: '896c088c-7468-4045-965b-e8ae134086c2', readOnly: true },
    ],
  },

  /* 
      If you can't answer questions on it, the assessment is complete.
      This typically happens for a couple reasons:
           - durationMinutes was set, and you ran out of time
           - the oldest late deadline, or due date if no late deadlines. 
           - PrairieTest says that the assessment is complete?
      The completion date can be different for different students.
   */

  afterComplete: {
    hideQuestions: {
      beforeDateEnabled: true,
      beforeDate: '2024-03-23T23:59',
      afterEnabled: true,
      afterDateEnabled: '2024-03-23T23:59', /* can only be set if beforeDate is set (UI limitation) */
    },
    hideScore: { beforeDateEnabled: true, beforeDate: '2024-03-23T23:59' },
  }
};

Another draft schema after discussing with @mwest1066 .

[reteps]

Further notes from discussion on 9/2 with @mwest1066

• Access control can be an object, or a list of objects.
• All overrides will happen at the key level -- you cannot override entire objects at once
  • Lists are overridden all at once. For every list in the schema, there will be a db-only key marking a distinction between list: [] and an unset list, e.g. late_deadlines_has_override. We need this key anyways since all lists will be stored in a separate table when they are normalized to the database.
• There is an implicit default access control rule that provides defaults for all options. This means that even the 'base' access control rule acts as an override of this default.
• There is no distinction between the first access control rule, and the other access control rules, other than the order that they are applied in to resolve the final configuration. This means that for configuration, we can have a single accessControl: {} | [{}] field.
• We should NEVER use NULL to represent a special value. It should only represent 'UNSET'. For example, we wouldn't want a NULL due date to mean that something doesn't have a due date, because we are using NULL to represent an override, and this would cause issues distinguishing the two. Thus:

Every field where not setting it means something either needs:

• explicit column in database
• explicit enabled flag for that value in schema

For the user interface, we want the ability to toggle on/off a setting without losing the value of that setting. For example, we don't want dates, strings, numbers to be lost. Thus the JSON schema will have this rule:

every non-boolean configurable field has a true/false in for enabling, if it can be enabled/disabled

This has the downside of allowing 'nonsense' JSON schema configurations (e.g. a configured property with enabled: false, or afterEnabled: true without beforeEnabled: true), but this is acceptable as we move towards a UI-first approach. We should do a good job documenting this behavior in the case of manual JSON edits.