tvalentyn

our use case is compatibility testing. We do use [test] extras, but they add a range, for example 'pyarrow>=3.0.0,

Are you referring to the suggestion by masenf@ or there is a simpler example of what that could look like?

You can make the generated transforms more discoverable by importing them in `__init.py__` files. that might generating an additional file(s) `_external_transforms.py`, and importing _external_transforms.py in `__init__.py` . That would make...

> What if instead they were in a single subdirectory (like with protos) I actually also thought about it and hesitated to suggest only since it felt a bit late...

i think my comments were addressed. LGTM from my side as long as tests pass.

actually, i tried running `gradlew generateExternalTransformWrappers` in a clean environment and it failed. I think the intent of that command should be to regenerate the yaml portion only, but not...

you could consider extracting the portion that generates the yaml into a separate script. If you keep the script in one file that can be fine too, but then generating...

> I think we need to put the import back in the try:, except: pass block? That will work, alternatively, you can have dedicated functions: to generate config, and to...

> t. The external_transform environment tests would fail if there is an additional staging file by default. Can we stage job submission dependencies without including them in the runtime environment...

Hey @riteshghorse , could we extract the commits that log the runtime dependencies and merge those before next release cut while submission dependencies portion is being sorted out? Thanks!