TheRacetrack
Isolate job's requirements
A job runs a single Python interpreter process that contains 2 codebases:
- wrapper code (handling http, metrics, swagerUI, etc)
- actual job's logic
Each of them have its third-party dependencies installed with pip install -r requirements.txt.
That raises a risk of conflicting packages.
Although they are installed in 2 different virtual environments, they're still running in the same single Python process that makes it impossible to load 2 different versions of the same package at the same time.
For instance, if wrapper code relies on fastapi==0.100.0, but the job depends on fastapi==0.90.0, there might be the clash, cause the wrapper's library takes precedence when loading modules in Python and the job will use fastapi==0.90.0
I see the only good way out of here: Wrapper code shouldn't have any third-party dependency (that could be overwritten or conflicted by a job). Then, there won't be any conflict. It can either implement the necessary logic on its own or use "forked" versions of libraries (eg. FastAPI renamed to fastapi_racetrack package).
This task has a low priority as it rarely happens to break something (but when it does, it's astonishing and hard to find).
This issue popped up with Pydantic (ours is ==2.6.1 atm) and a certain Walking Cat (theirs is not 2.6.1) when using the Dockerfile jobtype, so we might want to look into vendoring Pydantic.
Speaking of "vendoring", that might be a good workaround. What we really need is to create a copy of pydantic package with all imports updated so that it can be imported with
from pydantic_vendored import ... apart from catwalk's pydantic module.
Do you know a tool for doing this?
The more I read and think about vendoring, the less I like it. We are taking on the responsibilities of maintaining our vendored package with all that entails.
I'd rather look for a way to isolate Catwalk from racetrack when using the Dockerfile Job Type.
Wrapper code shouldn't have any third-party dependency
Difficult but clean.
a certain Walking Cat
Why is Catwalk being developed as a RT job?
look into vendoring Pydantic
Why? Are you thinking of having different versions of pydantic, one for RT and one for the job, and then loading the RT one from a different location?
Concerning vendoring: it's either bad nor good, it's situational. It doesn't feel good in Python because of python's tooling which is at this stage not very modern. In golang for example, vendoring is clean and efficient, and supported in the first-party toolchain. go mod vendor -u should tell you all you need to know.
I'm not sure we should accommodate concurrent different versions of the same library in a jobtype. In the ideal scenario, there's no third party dependencies in the job type plugin. Otherwise, if someone wants to use a library the jobtype uses, the docs should be clear they can only use the version shipped with the jobtype, and maybe even the builder should error out if the user has requested a library shipped in the jobtype. So a good way to accomplish this is like Irek's first thought, to remove third party dependencies.
My concern is, there's many people who have looked into doing this with Python over the last couple of decades, and no one has a good solution. The best I've seen is having 2 versions installed, but not having 2 versions active in the same running interpreter process. I just think if we try to do this we'll end up with all sorts of really nasty errors to troubleshoot because we're doing something Python is not designed for, and thinking we're more clever than so many other people.
And speaking of jobtypes, isn't this an issue for the Python jobtype, not for core RT?
Why is Catwalk being developed as a RT job?
It's a bit different. Their RT job requires Catwalk as one of dependencies in requirements list. To sum up: Wrapper code (which uses pydantic) imports a job, a job may import Catwalk, which in turn imports a different version of pydantic. That's where the conflict comes from.
Why? Are you thinking of having different versions of pydantic, one for RT and one for the job, and then loading the RT one from a different location?
Yes, that's exactly why.
Their RT job requires Catwalk as one of dependencies in requirements list
Ah, Catwalk client. Got it.
Still feels like your initial thought ot removing third party dependencies from the job type is the most viable option. Actually should be a strong recommendation for job type developers.
And speaking of jobtypes, isn't this an issue for the Python jobtype, not for core RT?
You're right, this issue originated from Racetrack back in the days, but at this moment it's more related to https://github.com/TheRacetrack/job-runner-python-lib
When I look at its code it has a few occurences of pydantic usage. This library originated from a racetrack client and now it's a small subset of Racetrack's codebase. I suppose it should be fairly easy to replace pydantic with something built-in (like dataclasses). That would start a process of removing third-party dependencies from job-runner-python-lib. It also means that its codebase would start its own life (not longer being shared with Racetrack core).
This is a general discussion, so I've made a separate issue for pydantic removal: https://github.com/TheRacetrack/job-runner-python-lib/issues/3