add `System.vendor_deleted_date` db model field; switch DB engines to use custom JSON serialization to handle non-standard data types

[adamsachs]

Closes PROD-1979

Description Of Changes

Adds in DB model support for a new vendor_deleted_date field on the System model, added in corresponding fideslang model as a potential means of supporting the GVL vendor deleted date field.

As a side effect of this change, a new date field is now needs to be stored in a JSONB field (in the SystemHistory table), we needed to ensure that those fields were being properly encoded (serialized) as JSON to store in the DB. Instead of special-casing this particular field, we decided to update our SQLAlchemy DB engines more broadly to use json (de)serializer functions that leverage our CustomJSONEncoder that handles datetimes and a few other data types that cannot be serialized to JSON via json.dumps().

Corresponding upstream PRs (should be merged first):

• https://github.com/ethyca/fideslang/pull/10
• https://github.com/ethyca/fides-services/pull/134 (not strictly required to be merged first)

Corresponding downstream PR:

• https://github.com/ethyca/fidesplus/pull/1392

Code Changes

• [x] update DB model
• [x] add column with DB migration
• [x] update common DB engines to use CustomJSONEncoder for JSON (de)serialization
• [x] update (remove) previous manual uses of CustomJSONEncoder for DB fields to leverage the now generic/automatic handling via the DB engine

Still TODO:

• [x] populate the property on non-bulk compass vendor add?

Steps to Confirm

• [x] tested full stack in corresponding plus PR: https://github.com/ethyca/fidesplus/pull/1392

Pre-Merge Checklist

• [x] All CI Pipelines Succeeded
• Documentation:
  • [ ] documentation complete, PR opened in fidesdocs
  • [ ] documentation issue created in fidesdocs
• [x] Issue Requirements are Met
• [ ] Relevant Follow-Up Issues Created
• [x] Update CHANGELOG.md
• [ ] For API changes, the Postman collection has been updated

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
fides-plus-nightly	⬜️ Ignored (Inspect)	Visit Preview		May 8, 2024 2:17am

[cypress[bot]]

Passing run #7629 ↗︎

0

4

0

0

0

Details:

Merge aaea874cd1491c2d96f2b5342a0ee408918a78b6 into 0e27b2189e97b98347de649c33d3...
Project: fides	Commit: 57c7d641c1 ℹ️
Status: Passed	Duration: 00:35 💡
Started: May 8, 2024 2:29 AM	Ended: May 8, 2024 2:29 AM

Review all test suite changes for PR #4818 ↗︎

[adamsachs]

ok @pattisdr i think the update to use the custom encoder at the DB engine level is looking good, per our discussion yesterday! (tests seem to be passing).

would like to get your eyes/thoughts on this before moving on further though!

[pattisdr]

Looking @adamsachs!

[pattisdr]

lots of failing DSR 3.0 tests 😬

[adamsachs]

lots of failing DSR 3.0 tests 😬

whoops, the tests i ran locally were far too narrow :( i'll take a closer look, sorry for asking prematurely!

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 98.46154% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 86.79%. Comparing base (1e180ec) to head (eeda81e).

Files	Patch %	Lines
src/fides/api/task/execute_request_tasks.py	50.00%	1 Missing :warning:

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4818   +/-   ##
=======================================
  Coverage   86.78%   86.79%           
=======================================
  Files         346      347    +1     
  Lines       20919    20932   +13     
  Branches     2734     2734           
=======================================
+ Hits        18154    18167   +13     
  Misses       2289     2289           
  Partials      476      476           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[adamsachs]

The Admin UI types/code changes are pretty solid. I didn't UAT the UI though so you should just grab a quick screenshot and get some eyes from @Kelsey-Ethyca or otherwise to see, since that System form is already pretty monstrous

for UAT purposes, here are some screenshots of the new field on my local:

• when creating a system, field is populated from compass:

• when creating a system, field is not populated from compass (vendor doesn't have a deleted_date):

• editing a saved system with the field populated (field is not editable):

• editing a saved system with the field not populated (field is still not editable):

[adamsachs]

merging, discussed with @Kelsey-Ethyca that she will UAT once it's on main and this is a relatively low-risk update in any case 👍