Generic Async Callback Integration Support

[pattisdr]

Closes #PROD-2032

Description Of Changes

Adds DSR 3.0 support for a custom SaaS connector that can't fulfill an access or erasure request or immediately, but can post the results back to Fides when the action is complete.

Code Changes

• [x] Adds new saas config support for a callback async strategy
• [x] Adds the ability to inject reply to and reply to token headers that describe how to send async results back to Fides
• [x] Adds a new callback endpoint POST /request-task/callback which can take in access results as a list of rows or rows masked (integer) can be supplied, or an empty json request body is sufficient if you don't want to pass back any results, and instead just verify completion.

Steps to Confirm

• [x] DSR 3.0 must be enabled
• [x] Create a custom connector to a third party service with the following on the API endpoints:

          async_config:
            strategy: callback
          headers:
            - name: reply-to
              value: <reply_to>
            - name: reply-to-token
              value: <reply_to_token>

• [x] Demonstrate Fides putting request task in paused if async callback strategy denoted. Verify request to third party includes reply-to and reply-to tokens
• [x] Verify RequestTask can be resumed by posting to reply-to url with reply-to-token

• See saas connector tests for example

Pre-Merge Checklist

• [ ] All CI Pipelines Succeeded
• Documentation:
  • [x] documentation complete, https://github.com/ethyca/fidesdocs/pull/361
• [x] Issue Requirements are Met
• [x] Relevant Follow-Up Issues Created
• [x] Update CHANGELOG.md

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
fides-plus-nightly	⬜️ Ignored (Inspect)	Visit Preview		May 22, 2024 8:11pm

[cypress[bot]]

Passing run #7874 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge a9ae9fd387e106917ae05c3f31e7258eaf73e410 into bb85e07fd29fee8418ef3d038067...
Project: fides	Commit: 1f74df7d09 ℹ️
Status: Passed	Duration: 00:34 💡
Started: May 22, 2024 8:22 PM	Ended: May 22, 2024 8:22 PM

Review all test suite changes for PR #4865 ↗︎

[pattisdr]

@galvana hoping to get your feedback here when you have a chance! ~No tests yet~, just want feedback on approach

OK test coverage added!

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 96.63866% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 86.80%. Comparing base (bb85e07) to head (a9ae9fd).

Files	Patch %	Lines
src/fides/api/oauth/utils.py	84.00%	2 Missing and 2 partials :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4865      +/-   ##
==========================================
+ Coverage   86.72%   86.80%   +0.08%     
==========================================
  Files         347      347              
  Lines       21121    21219      +98     
  Branches     2765     2783      +18     
==========================================
+ Hits        18317    18420     +103     
+ Misses       2315     2311       -4     
+ Partials      489      488       -1     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[pattisdr]

@galvana thank you so much for your review, requested changes made. Failing saas connectors seem related to marigold and maybe hubspot, unrelated to this work.

[pattisdr]

Converting back to draft ...

[pattisdr]

Some manual testing of the changes to return a status of "awaiting_processing" instead of "paused" for async tasks awaiting a callback.

Privacy Request Tasks returns tasks awaiting an async callback with status: "awaiting_processing"

GET Privacy Request Tasks http://localhost:8080/api/v1/privacy-request/pri_475123ed-2a21-41fb-975e-0a3908580549/tasks

[
  {
    "id": "req_39f469b4-0cc2-45ca-a77d-e97563d91e53",
    "collection_address": "__ROOT__:__ROOT__",
    "status": "complete",
    "created_at": "2024-05-21T22:20:27.036967+00:00",
    "updated_at": "2024-05-21T22:20:27.036967+00:00",
    "upstream_tasks": [],
    "downstream_tasks": [
      "test_system_async_callback_example_api:user"
    ],
    "action_type": "access"
  },
  {
    "id": "req_f35c5e1e-456c-42f4-b4fa-baffde175f74",
    "collection_address": "test_system_async_callback_example_api:user",
    "status": "awaiting_processing",
    "created_at": "2024-05-21T22:20:27.059852+00:00",
    "updated_at": "2024-05-21T22:20:27.137950+00:00",
    "upstream_tasks": [
      "__ROOT__:__ROOT__"
    ],
    "downstream_tasks": [
      "__TERMINATE__:__TERMINATE__"
    ],
    "action_type": "access"
  },
  {
    "id": "req_fa6e5518-dfeb-4274-9821-64bca2f20961",
    "collection_address": "__TERMINATE__:__TERMINATE__",
    "status": "pending",
    "created_at": "2024-05-21T22:20:27.063794+00:00",
    "updated_at": "2024-05-21T22:20:27.063794+00:00",
    "upstream_tasks": [
      "test_system_async_callback_example_api:user"
    ],
    "downstream_tasks": [],
    "action_type": "access"
  }
]

Verbose Privacy Request Status has embedded execution logs with "awaiting_processing"

Awaiting_processing is saved as "paused" in the database

Avoiding migration by renaming unused "paused" status in the application, but leaving as "paused" in the db

fides=# select status from executionlog;
    status     
---------------
 in_processing
 paused
(2 rows)

fides=# select status from requesttask;
  status  
----------
 complete
 pending
 paused
(3 rows)

Getting Privacy Request Execution Logs directly also uses "awaiting_processing"

http://localhost:8080/api/v1/privacy-request/pri_475123ed-2a21-41fb-975e-0a3908580549/log?page=1&size=50

{
  "items": [
    {
      "collection_name": "user",
      "fields_affected": [],
      "message": "starting",
      "action_type": "access",
      "status": "in_processing",
      "updated_at": "2024-05-21T22:20:27.091181+00:00",
      "connection_key": "test_system_async_callback_example_api",
      "dataset_name": "test_system_async_callback_example_api"
    },
    {
      "collection_name": "user",
      "fields_affected": [],
      "message": "",
      "action_type": "access",
      "status": "awaiting_processing",
      "updated_at": "2024-05-21T22:20:27.103386+00:00",
      "connection_key": "test_system_async_callback_example_api",
      "dataset_name": "test_system_async_callback_example_api"
    }
  ],
  "total": 2,
  "page": 1,
  "size": 50,
  "pages": 1
}