Pants not picking up plugin code changes without force restarting pantsd

[ndellosa95]

Describe the bug Ever since upgrading to 2.19.x I've been seeing pantsd be super sticky when working on plugin code, where changes won't be picked up unless I force restart the scheduler by running rm -rf .pants.d. This stickiness is even persisting across CI runs, which I'm guessing has to do with the way pantsbuild/actions/init-pants@v4-scie-pants sets up Pants.

Pants version 2.19.1

OS Both my intel mac and the intel linux our CI runs on.

Additional info Add any other information about the problem here, such as attachments or links to gists, if relevant.

[ndellosa95]

Upon further investigation, the CI issue seems to be a caching issue actually. I added a --print-stacktrace to the CI step that was failing and it succeeded. Then I tried changing the cache key and running rm -rf .pants.d on CI without --print-stacktrace and it failed again. When I set remote_cache_read = false it then succeeded without needing --print-stacktrace. So it seems that the remote caching is not taking the changed plugin code into effect.