[CI] Fix flaky CI failures

[RRosio]

• [ ] end_to_end/test_display_isolation.py
• [ ] end_to_end/test_save_readonly_as.py
• [ ] reducing the number of Python versions tested might help mitigate flakiness if it is caused by resource limitations