fix(marshmallow): add compatibility layer for Flask-AppBuilder with marshmallow 4.x

[kevin-lann]

SUMMARY

Upgrades marshmallow version to >= 4.0

• Flask-AppBuilder 5.0.0 auto-generates schema fields for SQLAlchemy relationships that don't exist in marshmallow 4.x's stricter field validation, causing KeyError during schema initialization.
• Our solution patches marshmallow's Schema._init_fields method to detect missing fields that Flask-AppBuilder 5.0.0 auto-generates, and then adds these missing fields dynamically as raw fields with appropriate flags & retry initialization until all missing fields are resolved.
• So, for example, it can handle multiple fields in a sequence (permission_id, view_menu_id, db_id, chart_id, dashboard_id, user_id). All of this logic for this bullet point is encapsulated in the marshmallow_fix.py module, whose job is to specifically address the Flask-AppBuilder 5.0.0 incompatibility with marshmallow 4.x, and then app.py applies the patch by importing it from marshmallow_fix.py.
• All changes in files other than app.py and marshmallow_fix.py are minor convention/syntax changes that are required for the Marshmallow upgrade to >= 4.0.0 (See https://marshmallow.readthedocs.io/en/stable/upgrading.html for more info)

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

BEFORE: Upon pinning marshmallow>=4 in pyproject.tomland initializing the superset app, an initialization error occurs.

AFTER: App runs without error

TESTING INSTRUCTIONS

Start superset application, see that no marshmallow related errors show up.

Run tests with docker compose run --rm superset python -m pytest tests/unit_tests/test_marshmallow_compatibility.py -v

Unit tests created:

1. test_patch_marshmallow_for_flask_appbuilder_applies_patch - Verifies the patch function correctly modifies the Schema class
2. test_patch_functionality_with_real_schema_creation - Tests actual schema creation works with the patch
3. test_patch_handles_schema_with_no_fields - Ensures empty schemas work properly
4. test_raw_field_creation_and_configuration - Validates Raw field configuration
5. test_print_function_can_be_mocked - Tests logging functionality can be tested
6. test_keyerror_exception_handling - Verifies KeyError exception handling
7. test_schema_declared_fields_manipulation - Tests field manipulation capabilities
8. test_flask_appbuilder_field_names_list - Validates the field names our fix handles
9. test_patch_function_is_callable - Ensures the patch function works correctly
10. test_marshmallow_schema_basic_functionality - Confirms basic marshmallow functionality still works

ADDITIONAL INFORMATION

• [x] Has associated issue: Fixes issue #33162
• [ ] Required feature flags:
• [ ] Changes UI
• [ ] Includes DB Migration (follow approval process in SIP-59)
  • [ ] Migration is atomic, supports rollback & is backwards-compatible
  • [ ] Confirm DB migration upgrade and downgrade tested
  • [ ] Runtime estimates and downtime expectations provided
• [ ] Introduces new feature or API
• [ ] Removes existing feature or API

Credit to

@Austin-X @Eyang0612 @PDOracle @MasahisaSekita

[korbit-ai[bot]]

Based on your review schedule, I'll hold off on reviewing this PR until it's marked as ready for review. If you'd like me to take a look now, comment /korbit-review.

Your admin can change your review schedule in the Korbit Console

[kevin-lann]

/korbit-review

[kevin-lann]

@mistercrunch Hi, could you re-run workflows on this PR? Also lmk if you have any concerns/suggestions about the compatibility changes here. Much appreciated 🙏

[rusackas]

re-running workflows 🤞

[rusackas]

CC @dpgaspar @Antonio-RiveroMartnez

[kevin-lann]

@rusackas The tests/checks should be all fixed now (hopefully). Could you rerun once again?

[codecov[bot]]

Codecov Report

:x: Patch coverage is 77.27273% with 10 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 67.55%. Comparing base (1e4bc6e) to head (33c0814). :warning: Report is 479 commits behind head on master.

Files with missing lines	Patch %	Lines
superset/marshmallow_compatibility.py	78.26%	2 Missing and 3 partials :warning:
superset/themes/schemas.py	70.00%	3 Missing :warning:
superset/app.py	60.00%	2 Missing :warning:

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #35920       +/-   ##
===========================================
+ Coverage        0   67.55%   +67.55%     
===========================================
  Files           0      637      +637     
  Lines           0    46847    +46847     
  Branches        0     5084     +5084     
===========================================
+ Hits            0    31646    +31646     
- Misses          0    13912    +13912     
- Partials        0     1289     +1289     

Flag	Coverage Δ
hive	43.77% <75.00%> (?)
mysql	67.10% <77.27%> (?)
postgres	67.15% <77.27%> (?)
presto	47.36% <75.00%> (?)
python	67.51% <77.27%> (?)
sqlite	66.77% <77.27%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kevin-lann]

The above commit should fix the remaining checks https://github.com/apache/superset/pull/35920/commits/33c081401717453b5311c254052a6da21dc6c819