compat: Remove direct dependency on pandas

[hoxbro]

Really early. This is to test-run our testsuite without having Pandas as a direct dependency.

[philippjfr]

Great start, though we need to work on removing the dependency in bokeh first.

[codecov[bot]]

Codecov Report

:x: Patch coverage is 76.31579% with 18 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 81.95%. Comparing base (f9a72b7) to head (11dd401).

Files with missing lines	Patch %	Lines
panel/tests/io/test_cache.py	64.28%	5 Missing :warning:
panel/tests/io/test_location.py	50.00%	3 Missing :warning:
panel/tests/pane/test_vega.py	84.61%	2 Missing :warning:
panel/tests/test_custom.py	60.00%	2 Missing :warning:
panel/tests/test_expression.py	33.33%	2 Missing :warning:
panel/tests/widgets/test_tables.py	33.33%	2 Missing :warning:
panel/pane/vizzu.py	50.00%	1 Missing :warning:
panel/tests/widgets/test_tqdm.py	0.00%	1 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8236      +/-   ##
==========================================
- Coverage   86.02%   81.95%   -4.07%     
==========================================
  Files         348      348              
  Lines       54437    54475      +38     
==========================================
- Hits        46827    44647    -2180     
- Misses       7610     9828    +2218     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.