Another way to test packages in dev stage

[frankchen211]

Allow resolving a package from a random specified place instead of paths from rezconfig packages_path for testing purpose before release to production repo.

Let's say we have a package helloworld in d:\dev\helloworld (no version specified at this time) want to test, hope to init env like this: rez-env helloworld pkg1 pkg2 ... -d d:\dev helloworld will be resolved to D:\dev\helloworld, pkg1 and pkg2 resolved to a repository path in rez config. or if they are also in D:\dev folder.

helloworld                d:\dev\helloworld
pkg1-3.3                   c:\users\user\packages\pkg1\3.3\                                           (local)
pkg2-1.0.0                c:\users\user\packages\pkg2\1.0.0                                          (local)

or just directly specify the package path: rez-env helloworld@d:\dev\helloworld pkg1 pkg2 ...

Motivation Current workflow for testing a package generally is rez-build -i, then rez-test if we have unit test code or open dcc or rez-env xxx -- python to test. This means you have to copy a package to the local package path , this path also in packages_path in rezconfig.py to search pacakges by default.

Reason why I don't test a packge in this way: In my studio, all our workstations will be used to render jobs at night, which means if I have some packages are in test stage in local, any production render jobs that using my workstation will use these code, this is not what we are expected. (Some packages we do mirror to local to speed up the load time in production, local path is not always for testing purpose.)

If I want to invite other developers/artists to test my packages(not a formal test in our pipeline), I have to copy to a central repository and give a testing version carefully to avoid affecting production, since these packages are in develop stage, I really don't want to release it to production repository, instead just putting it to a sandbox and tell rez-env to resolve these packages from given paths, required pacakges still can be resolved from formal production repos.

[ColinKennedy]

Without getting too deep into pipeline specifics, because testing / deployment is often less a question for how Rez works and more how pipelines choose to adopt Rez (e.g. do artists press a button in a GUI to load Rez, do they write rez-env directly in the command-line, etc). Let's look at how to handle this with Rez, assuming no pipeline.

Assuming you want to develop / test a package called "package_foo"

• git clone the repository of package_foo
• cd {repository_root}/path/to/package_foo
• Bump the major / minor / patch version of package_foo's package.py to a new version
  • Notice: I do not use a testing version. I just bump one of the {major}.{minor}.{patch} tokens. The reason why is because of the next couple steps, below.
• build to a network location, visible to users and the farm (e.g. /path/to/network)
  • e.g. rez-build --clean --install --prefix=/path/to/network
• Then I'd create a context which represents "my patched package_foo + all of the other released packages"

REZ_PACKAGES_PATH=/path/to/network:`rez-config release_packages_path` \
    rez-env package_foo==the.version.here some_pakages-1 that_users-2 would_want-1 \
    --output /path/to/network_contexts/context.rxt

This creates a context which is just my package_foo's isolated changes but also includes all of the other released packages which I have not changed.

And then I'd either...

• Ask users to test out my changes by loading the context, via rez-env --input /path/to/network_contexts/context.rxt
• Use rez-bundle to produce a bundle for that context.rxt, and then ask users to run tools through that

Although in reality, I actually wouldn't use either option because when I worked Rez at Animal Logic, they had a GUI that in many ways abstracted this workflow. But this is how I would do it in a vanilla Rez environment, assuming no pipeline. Is there something in that suggested workflow I just described that you think needs improvement?

Side note: A key point of this workflow is that "/path/to/network" is a unique path name. Usually a real path would include a ticket number, like /path/to/network/PIPE-1234-my_testing_changes). Because the paths are unique, there's no chance of accidentally mixing your changes with other's work or with your past work. I made a video tutorial related to this topic. It may be good as reference: https://www.youtube.com/watch?v=rPe6JpzE_Nc. It's about automating developer environments and ties into this topic pretty well.

[nerdvegas]

I would strongly suggest that your render jobs be configured to use a different REZ_PACKAGES_PATH setting that does not include $home/packages. If you've implemented some form of package localisation (as it sounds like), then I would make this separate to $home/packages. To me this sounds like the real source of your problems - you're unable to differentiate between these two quite different use cases.

Hth A

On Tue, Mar 29, 2022 at 5:58 AM Colin Kennedy @.***> wrote:

Without getting too deep into pipeline specifics, because testing / deployment is often less a question for how Rez works and more how pipelines choose to adopt it (e.g. do artists press a button in a GUI, write in the command-line, etc). Let's look at how to handle this with Rez, assuming no pipeline.

Assuming you want to develop / test a package called "package_foo"

• git clone the repository of package_foo
• cd {repository_root}/path/to/package_foo
• Bump the major / minor / patch version of package_foo's package.py to a new version
• build to a network location, visible to users and the farm (e.g. /path/to/network)
  • e.g. rez-build --clean --install --prefix=/path/to/network

Then I'd create a context which represents "my patched package_foo + all of the other released packages"

• REZ_PACKAGES_PATH=/path/to/network:rez-config release_packages_path rez-env package_foo==the.version.here some_pakages-1 that_users-2 would_want-1 --output /path/to/network_contexts/context.rxt
  • This creates a context which is just my package_foo's isolated changes but also includes all of the other released packages which I have not changed.

And then I'd either...

• Ask users to test out my changes by loading the context, via rez-env --input /path/to/network_contexts/context.rxt
• Use rez-bundle https://github.com/nerdvegas/rez/wiki/Bundles to produce a bundle for that context.rxt, and then ask users to run tools through that

Although in reality, I actually wouldn't use either option because when I worked Rez at Animal Logic, they had a GUI that in many ways abstracted this workflow. But this is how I would do it in a vanilla Rez environment, assuming no pipeline. Is there something in that suggested workflow I just described that you think needs improvement?

Side note: I made a video tutorial related to this topic. It may be good as reference: https://www.youtube.com/watch?v=rPe6JpzE_Nc. It's about automating developer environments and ties into this topic pretty well.

— Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1263#issuecomment-1081023290, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMOUSRVN3PKB6BE3FB5KZLVCH6NJANCNFSM5RZ63D4A . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[frankchen211]

build to a network location, visible to users and the farm (e.g. /path/to/network)

• e.g. rez-build --clean --install --prefix=/path/to/network
• Then I'd create a context which represents "my patched package_foo + all of the other released packages"

REZ_PACKAGES_PATH=/path/to/network:`rez-config release_packages_path` \
    rez-env package_foo==the.version.here some_pakages-1 that_users-2 would_want-1 \
    --output /path/to/network_contexts/context.rxt

Your solution here seems add the random new package path /path/to/network to search path and output a context file, which is similar to my proposal - make rez-env directly support this random path: rez-env package_foo some_pakages-1 that_users-2 would_want-1 -d /path/to/network

Or directly tell rez where to load ackage_foo: rez-env package_foo@/path/to/network some_pakages-1 that_users-2 would_want-1 In this case, we don't need extra work any more:

• run rez-build, we don't use this cmd too much since lots of pipeline tools/dev written in python don't need to be build, or plugins need to be built we will use other method to do that, this depends on developers.
• a script to generate or modify sth like the example here to create a context file

Although in reality, I actually wouldn't use either option because when I worked Rez at Animal Logic, they had a GUI that in many ways abstracted this workflow. But this is how I would do it in a vanilla Rez environment, assuming no pipeline. Is there something in that suggested workflow I just described that you think needs improvement? Yes we provide GUI to artist to use, but for random tests(before available to production), cli makes it easier and flexible.

Side note: A key point of this workflow is that "/path/to/network" is a unique path name. Usually a real path would include a ticket number, like /path/to/network/PIPE-1234-my_testing_changes). Because the paths are unique, there's no chance of accidentally mixing your changes with other's work or with your past work. I made a video tutorial related to this topic. It may be good as reference: https://www.youtube.com/watch?v=rPe6JpzE_Nc. It's about automating developer environments and ties into this topic pretty well.

Yes, we do use ticket system and branches(ticket nubmer) in our workflow, that's why we dont' use versions until all tests done and merge to master/main branch to trigger deploy process to push to production env, during deploy we give a version.

[frankchen211]

There's lots of scenarios that we want to test packages, this would make rezconfig complicated and need some extra settings or wrappers in pipeline to differentiate different use cases.

Let's say I have a package that is used in render jobs also, when submit from a test env, a render node should pick that dev package also.

To do so, there's no need more extra work and simple. rez-env foo bar-1 -d /path/to/sandbox/issue-xxx Or rez-env foo@/path/to/sandbox/issue-xxx bar-1

rez-env foo bar-1 -d /path/to/sandbox/issue-xxx will be resovled to foo@/path/to/sandbox/issue-xxx bar-1

then use REZ_USED_REQUEST to init the render job.

[nerdvegas]

To be clear, does /path/to/sandbox/issue-xxx contain a built package, or are you still suggesting that you want to refer to unbuilt packages this way? I'll assume standard built packages in my response.

I'm not really sold on this yet, because you can just do this instead:

]$ rez-build -i -p /sandbox/issue-123  # install foo-1.2.3
..then..
]$ rez-env --paths /sandbox/issue-123:/standard1:/standardN foo-1.2.3 pkgB
pkgC

Currently, syntax used in rez-env is exactly the same as package requirements in a package.py, and I think that's a good thing. By supporting explicit paths like this (ie your 'foo@/path` eg), you either break that correlation, or you allow packages to do this too. If packages can do this, you bet quick fixes referring to temp repo paths will end up in production, and I think that would cause more problems in the long run.

If your example of -d here means to say, "add this path to REZ_PACKAGES_PATH instead of me having to change my rezconfig", then that's already very similar to the existing --paths option. I'll grant you it is a bit more user friendly though (since you don't have to repeat the rest of the packages path), so if that is what you mean, and if you named it something like --add-pre-paths/--add-post-paths instead, then I wouldn't be opposed to adding that.

Thx A

On Tue, Mar 29, 2022 at 2:11 PM frankchen211 @.***> wrote:

There's lots of scenarios that we want to test packages, this would make rezconfig complicated and need some extra settings or wrappers in pipeline to differentiate different use cases.

Let's say I have a package that is used in render jobs also, when submit from a test env, a render node should pick that dev package also.

To do so, there's no need more extra work and simple. rez-env foo bar-1 -d /path/to/sandbox/issue-xxx Or rez-env foo@/path/to/sandbox/issue-xxx bar-1

— Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1263#issuecomment-1081354531, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMOUSRGMDRVOXZBNE3PPJTVCJYF5ANCNFSM5RZ63D4A . You are receiving this because you commented.Message ID: @.***>

[ColinKennedy]

@nerdvegas I thought about @frankchen211 situation a bit more and see where he's coming from. I experienced his problem at AL several times. I'll try to explain the problem and how I "solved" it, using the vanilla Rez stuff I'd mentioned earlier.

• package_foo has a bug
• I go through the process of bug-fixing, building to /path/to/network etc as mentioned in my previous reply
  • To do this fix, I bumped the patch version of package_foo from 1.2.3 to 1.2.4
• Another developer comes and releases 1.3.0 to the same release path I was using as a fallback path for non-built packages.

Now I get a chat message from a tester "Colin, your bug fix is gone / doesn't work". It could be that the bug fix actually didn't work. But it's also possible that someone released over top of my change (as in this case)

Since I don't know which situation it is, every time I get a message like that, I have to check if that the user was getting my network-built path first. It's a bit tedious.

There's 2 ways Rez can "solve" this problem

• Replace the user's rez-env request
• Upversion your package to something very high

Replace the user's rez-env request

If the deployed / production rez-env request is rez-env package_foo-1.1+<2 another_package-2+ etc-3, I could give the user rez-env package_foo==1.2.4 another_package-2+ etc-3. That way, even if package_foo 1.3.0, 1.4.0, etc is released, there's no chance that Rez will pick those versions. My patch is always chosen.

However, this assumes that you even can modify the request, which may not be feasible. e.g. if your pipeline doesn't have a way of sparsely changing the version of a single package, you're out of luck. Which leads to the other solution...

Upversion your package to something very high

If I change package_foo's version to 1.999.0, it doesn't matter if 1.3.0, 1.4.0, etc come around because my version, which is actually just a bug fix + patch, will always be chosen. This is neither semantic nor reliable, because the request might actually be package_foo-1.1+<1.100.0 (just as an example).

If we had a means of telling Rez "only consider the packages built to this directory but fall back to the other paths for the Rez package families which aren't built", that might be a nice idiom to have.

[nerdvegas]

But if you had a means of telling rez that, then you're changing the request anyway (eg you're adding --add-pre-paths or whatever), and if you're changing the request, you may as well just specify your testing version explicitly..?

This idea has come up in a different form before - users are sometimes surprised when their locally installed package doesn't get used, because someone happened to release a higher version of the same package to their central repo.

"""If we had a means of telling Rez "only consider the packages built to this directory but fall back to the other paths for the Rez package families which aren't built", that might be a nice idiom to have."""

What's this about resolving to unbuilt packages again? A

On Tue, Mar 29, 2022 at 4:32 PM CrimsonScholar @.***> wrote:

@nerdvegas https://github.com/nerdvegas I thought about @frankchen211 https://github.com/frankchen211 situation a bit more and see where he's coming from. I experienced his problem at AL several times. I'll try to explain the problem and how I "solved" it, using the vanilla Rez stuff I'd mentioned earlier.

• package_foo has a bug
• I go through the process of bug-fixing, building to /path/to/network etc as mentioned in my previous reply
  • To do this fix, I bumped the patch version of package_foo from 1.2.3 to 1.2.4
• Another developer comes and releases 1.3.0 to the same release path I was using

Now I get a chat message from a tester "Colin, your bug fix is gone / doesn't work". It could be that the bug fix actually didn't work. But it's also possible that someone released over top of my change (as in this case)

Since I don't know which situation it is, every time I get a message like that, I have to check if that the user was getting my network-built path first. It's a bit tedious.

There's 2 ways Rez can "solve" this problem

• Replace the user's rez-env request
• Upversion your package to something very high

Replace the user's rez-env request

If the deployed / production rez-env request is rez-env package_foo-1.1+<2 another_package-2+ etc-3, I could give the user rez-env package_foo==1.2.4 another_package-2+ etc-3. That way, even if package_foo 1.3.0, 1.4.0, etc is released, there's no chance that Rez will pick those versions. My patch is always chosen.

However, this assumes that you even can modify the request, which may not be feasible. e.g. if your pipeline doesn't have a way of sparsely changing the version of a single package, you're out of luck. Which leads to the other solution... Upversion your package to something very high

If I change package_foo's version to 1.999.0, it doesn't matter if 1.3.0, 1.4.0, etc come around because my version, which is actually just a bug fix

• patch, will always be chosen. This is neither semantic nor reliable, because the request might actually be package_foo-1.1+<1.100.0 (just as an example).

If we had a means of telling Rez "only consider the packages built to this directory but fall back to the other paths for the Rez package families which aren't built", that might be a nice idiom.

— Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1263#issuecomment-1081421770, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMOUSW2ZCIAZT4JOEBX6PLVCKIW5ANCNFSM5RZ63D4A . You are receiving this because you were mentioned.Message ID: @.***>

[ColinKennedy]

if you're changing the request, you may as well just specify your testing version explicitly..?

That's assuming that you can apply your change sparsely. If you're trying to make a test environment that's "production + your one change", depending on how your pipeline is set up, the raw Rez request could look like this under the hood.

rez-env package_foo-1.1+<2 another_package-2+ etc-3

My one change, in this case, is "I want to change the package request to a locked patch". But your pipeline doesn't allow you to do that sparsely. So what choice do you have but to copy the whole request and edit the one package.

package_foo-1.1+<2 another_package-2+ etc-3 -> **package_foo==1.2.4** another_package-2+ etc-3
# alternatively you could just append your version to the end of the request
package_foo-1.1+<2 another_package-2+ etc-3 -> package_foo-1.1+<2 another_package-2+ etc-3 **package_foo==1.2.4**

It's the same effect either way - you're either replacing package_foo or adding an extra package_foo with the explicit version. But the problem I'll describe next applies in either approach.

But What If The Original Request Changes

Now another developer comes along and, in the production tool / bundle / w.e introduces a new change to the request. Maybe they add / remove a package or change a version range.

package_foo-1.1+<2 another_package-2+ etc-3 **brand_new_package-1+**

Now my test request, which was **package_foo==1.2.4** another_package-2+ etc-3, doesn't include brand_new_package-1+. My test environment is no longer "production + your one change". It's "out of date production + your one change".

My bit about "sparsely" is that if you don't have a mechanism, whether by GUI, CLI, w/e - to change the one package_foo but inherit the request of the package requests, you end up hard-coding the whole request when setting up a test environment, which isn't great because things can go out of date.

but fall back to the other paths for the Rez package families which aren't built

Sorry, I'll clarify - I never meant to insinuate source / developer packages. I'm always talking about built / installed Rez packages.

"If we had a means of telling Rez "For all built packages in this directory, only consider those versions / family names. For all other paths which may contain built packages (e.g. release_packages_path), resolve those as normal.

The end behavior / goal is - the Rez packages that I built are always used even if there's a later version of a package. And I didn't have to change the request in order to get that behavior (because as mentioned in the paragraph above, modifying the request, as a workflow, has some rough corners)

[ColinKennedy]

Btw for context, this problem isn't that much of an issue in practice. Just re-releasing a greater version in response to changes is good enough in most cases. I'm mostly just curious if this chat could spawn a better workflow. But if not, that's alright, the current workflow works well enough in most cases.

[frankchen211]

Currently, syntax used in rez-env is exactly the same as package requirements in a package.py, and I think that's a good thing. By supporting explicit paths like this (ie your 'foo@/path` eg), you either break that correlation, or you allow packages to do this too. If packages can do this, you bet quick fixes referring to temp repo paths will end up in production, and I think that would cause more problems in the long run.

Yes, in my changs, it allows that, but this is only used for quick test from production side, in most cases, production use released versioned packages. Explicit paths may esay to reproduce and debug things?

If your example of -d here means to say, "add this path to REZ_PACKAGES_PATH instead of me having to change my rezconfig", then that's already very similar to the existing --paths option. I'll grant you it is a bit more user friendly though (since you don't have to repeat the rest of the packages path), so if that is what you mean, and if you named it something like --add-pre-paths/--add-post-paths instead, then I wouldn't be opposed to adding that. Thx A

I found env REZ_USED_PACKAGES_PATH will try this if this can ensure env from local to farm is constant.

Thx.

[fnaum]

Hi there,

I guess Colin and Allan covered quite well. I would say that vanilla rez + Colin recommendations should do the trick for your use case.

As mentioned by Colin we have seen from the early days the issue where you have a REZ_PACKAGES_PATH=/test_path/packages:/central/rez/packages and a package with a higher version than your test version is released to / central/rez/packages and is picked instead of your test package in /test_path/packages.

This is very common (and dangerous) in your CI workflows when you are testing a chain of rez dependencies.

A small example, you have bar that depends on foo

name: foo version: 1.5.3 (your test version)

name: bar requires = [ 'foo-1.5+<2']

And your build chain in the CI is to build foo-1.5.3 to a temp area /var/tmp/CI_job_123/ and then when you trigger the build of bar, you pass the modified REZ_PACKAGE_PATH=/var/tmp/CI_job_123/:/central/rez/packages

All good until someone releases foo-1.6 and now your tests are no longer testing what you intended, and you have false positives which is quite dangerous

Since we have complex CI tests build chains with 20+ interdependent packages, sometimes it is not easy to spot what is going on, especially when we are doing 50 releases a day.

The way we solved that years ago was implementing a flag to rez-build (and rez-test, rez-env) called *--force-local , *which forces the solver to use the versions of the packages in your REZ_LOCAL_PACKAGES_PATH ( /var/tmp/CI_job_123) , if it can get a solve with that local package version it fails the solve.

We also have another one more relaxed that is handy in other use cases called *--favour-local, *which as the name suggests if you have foo-1.5.3 in your /var/tmp/CI_job_123/ and someone releases foo-1.6 it will still prefer foo-1.5.3. But if the solver can not get a solve using that version it would happily take foo-1.6 from /central/rez/packages

If you think that this could be useful, we can take a look at how hard it is to extract that functionally (we have modified heavily rez-build and rez-test over the years) and push it upstream.

Cheers, Fede

On Thu, Mar 31, 2022 at 9:09 PM frankchen211 @.***> wrote:

Currently, syntax used in rez-env is exactly the same as package requirements in a package.py, and I think that's a good thing. By supporting explicit paths like this (ie your 'foo@/path` eg), you either break that correlation, or you allow packages to do this too. If packages can do this, you bet quick fixes referring to temp repo paths will end up in production, and I think that would cause more problems in the long run. Yes, in my changs, it allows that, but this is only used for quick test from production side and, in most cases, production use released versioned packages.

If your example of -d here means to say, "add this path to REZ_PACKAGES_PATH instead of me having to change my rezconfig", then that's already very similar to the existing --paths option. I'll grant you it is a bit more user friendly though (since you don't have to repeat the rest of the packages path), so if that is what you mean, and if you named it something like --add-pre-paths/--add-post-paths instead, then I wouldn't be opposed to adding that. Thx A

I found env REZ_USED_PACKAGES_PATH will try this if this can ensure env from local to farm is constant.

Thx.

— Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1263#issuecomment-1084366746, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVRQREUSVESP7XLGAZ54T3VCV2VJANCNFSM5RZ63D4A . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[nerdvegas]

Hey Fede,

Just FYI, I think the most elegant way to implement the --favour-local functionality you describe is probably with a new package orderer. And now I recall - the problem we ran into when discussing this ages ago, is that orderers can't be stacked. So that could interfere with orderers already in place.

The proper fix for this imo is -

• implement a new "stacked" package orderer, that can stack the result of multiple other orderers;
• use that to combine existing orderer with a favour-local orderer, if necessary;
• package that into a convenient --favour-local cli option / api func arg

I think this is the best approach because it uses existing mechanisms and doesn't introduce any special cases (besides the trivial addition of the --favour-local option).

A

On Fri, Apr 1, 2022 at 10:46 PM Federico Naum @.***> wrote:

Hi there,

I guess Colin and Allan covered quite well. I would say that vanilla rez + Colin recommendations should do the trick for your use case.

As mentioned by Colin we have seen from the early days the issue where you have a REZ_PACKAGES_PATH=/test_path/packages:/central/rez/packages and a package with a higher version than your test version is released to / central/rez/packages and is picked instead of your test package in /test_path/packages.

This is very common (and dangerous) in your CI workflows when you are testing a chain of rez dependencies.

A small example, you have bar that depends on foo

name: foo version: 1.5.3 (your test version)

name: bar requires = [ 'foo-1.5+<2']

And your build chain in the CI is to build foo-1.5.3 to a temp area /var/tmp/CI_job_123/ and then when you trigger the build of bar, you pass the modified REZ_PACKAGE_PATH=/var/tmp/CI_job_123/:/central/rez/packages

All good until someone releases foo-1.6 and now your tests are no longer testing what you intended, and you have false positives which is quite dangerous

Since we have complex CI tests build chains with 20+ interdependent packages, sometimes it is not easy to spot what is going on, especially when we are doing 50 releases a day.

The way we solved that years ago was implementing a flag to rez-build (and rez-test, rez-env) called *--force-local , *which forces the solver to use the versions of the packages in your REZ_LOCAL_PACKAGES_PATH ( /var/tmp/CI_job_123) , if it can get a solve with that local package version it fails the solve.

We also have another one more relaxed that is handy in other use cases called *--favour-local, *which as the name suggests if you have foo-1.5.3 in your /var/tmp/CI_job_123/ and someone releases foo-1.6 it will still prefer foo-1.5.3. But if the solver can not get a solve using that version it would happily take foo-1.6 from /central/rez/packages

If you think that this could be useful, we can take a look at how hard it is to extract that functionally (we have modified heavily rez-build and rez-test over the years) and push it upstream.

Cheers, Fede

On Thu, Mar 31, 2022 at 9:09 PM frankchen211 @.***> wrote:

Currently, syntax used in rez-env is exactly the same as package requirements in a package.py, and I think that's a good thing. By supporting explicit paths like this (ie your 'foo@/path` eg), you either break that correlation, or you allow packages to do this too. If packages can do this, you bet quick fixes referring to temp repo paths will end up in production, and I think that would cause more problems in the long run. Yes, in my changs, it allows that, but this is only used for quick test from production side and, in most cases, production use released versioned packages.

If your example of -d here means to say, "add this path to REZ_PACKAGES_PATH instead of me having to change my rezconfig", then that's already very similar to the existing --paths option. I'll grant you it is a bit more user friendly though (since you don't have to repeat the rest of the packages path), so if that is what you mean, and if you named it something like --add-pre-paths/--add-post-paths instead, then I wouldn't be opposed to adding that. Thx A

I found env REZ_USED_PACKAGES_PATH will try this if this can ensure env from local to farm is constant.

Thx.

— Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1263#issuecomment-1084366746, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ABVRQREUSVESP7XLGAZ54T3VCV2VJANCNFSM5RZ63D4A

. You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1263#issuecomment-1085800450, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMOUSX4YJGWU35TT32GC3DVC3OZRANCNFSM5RZ63D4A . You are receiving this because you were mentioned.Message ID: @.***>

[fnaum]

Hi Allan,

I'm 99% sure we implemented using the package orderer feature, and I think we never run into any problems with other packages orderers. I can check when I'm at work tomorrow.

F

On Sat, Apr 2, 2022 at 8:36 AM allan johns @.***> wrote:

Hey Fede,

Just FYI, I think the most elegant way to implement the --favour-local functionality you describe is probably with a new package orderer. And now I recall - the problem we ran into when discussing this ages ago, is that orderers can't be stacked. So that could interfere with orderers already in place.

The proper fix for this imo is -

• implement a new "stacked" package orderer, that can stack the result of multiple other orderers;
• use that to combine existing orderer with a favour-local orderer, if necessary;
• package that into a convenient --favour-local cli option / api func arg

I think this is the best approach because it uses existing mechanisms and doesn't introduce any special cases (besides the trivial addition of the --favour-local option).

A

On Fri, Apr 1, 2022 at 10:46 PM Federico Naum @.***> wrote:

Hi there,

I guess Colin and Allan covered quite well. I would say that vanilla rez

Colin recommendations should do the trick for your use case.

As mentioned by Colin we have seen from the early days the issue where you have a REZ_PACKAGES_PATH=/test_path/packages:/central/rez/packages and a package with a higher version than your test version is released to / central/rez/packages and is picked instead of your test package in /test_path/packages.

This is very common (and dangerous) in your CI workflows when you are testing a chain of rez dependencies.

A small example, you have bar that depends on foo

name: foo version: 1.5.3 (your test version)

name: bar requires = [ 'foo-1.5+<2']

And your build chain in the CI is to build foo-1.5.3 to a temp area /var/tmp/CI_job_123/ and then when you trigger the build of bar, you pass the modified REZ_PACKAGE_PATH=/var/tmp/CI_job_123/:/central/rez/packages

All good until someone releases foo-1.6 and now your tests are no longer testing what you intended, and you have false positives which is quite dangerous

Since we have complex CI tests build chains with 20+ interdependent packages, sometimes it is not easy to spot what is going on, especially when we are doing 50 releases a day.

The way we solved that years ago was implementing a flag to rez-build (and rez-test, rez-env) called *--force-local , *which forces the solver to use the versions of the packages in your REZ_LOCAL_PACKAGES_PATH ( /var/tmp/CI_job_123) , if it can get a solve with that local package version it fails the solve.

We also have another one more relaxed that is handy in other use cases called *--favour-local, *which as the name suggests if you have foo-1.5.3 in your /var/tmp/CI_job_123/ and someone releases foo-1.6 it will still prefer foo-1.5.3. But if the solver can not get a solve using that version it would happily take foo-1.6 from /central/rez/packages

If you think that this could be useful, we can take a look at how hard it is to extract that functionally (we have modified heavily rez-build and rez-test over the years) and push it upstream.

Cheers, Fede

On Thu, Mar 31, 2022 at 9:09 PM frankchen211 @.***> wrote:

Currently, syntax used in rez-env is exactly the same as package requirements in a package.py, and I think that's a good thing. By supporting explicit paths like this (ie your 'foo@/path` eg), you either break that correlation, or you allow packages to do this too. If packages can do this, you bet quick fixes referring to temp repo paths will end up in production, and I think that would cause more problems in the long run. Yes, in my changs, it allows that, but this is only used for quick test from production side and, in most cases, production use released versioned packages.

If your example of -d here means to say, "add this path to REZ_PACKAGES_PATH instead of me having to change my rezconfig", then that's already very similar to the existing --paths option. I'll grant you it is a bit more user friendly though (since you don't have to repeat the rest of the packages path), so if that is what you mean, and if you named it something like --add-pre-paths/--add-post-paths instead, then I wouldn't be opposed to adding that. Thx A

I found env REZ_USED_PACKAGES_PATH will try this if this can ensure env from local to farm is constant.

Thx.

— Reply to this email directly, view it on GitHub <https://github.com/nerdvegas/rez/issues/1263#issuecomment-1084366746 , or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ABVRQREUSVESP7XLGAZ54T3VCV2VJANCNFSM5RZ63D4A

. You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1263#issuecomment-1085800450, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAMOUSX4YJGWU35TT32GC3DVC3OZRANCNFSM5RZ63D4A

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1263#issuecomment-1086347517, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVRQRHSCXLTNMZFPLSYVWDVC5T57ANCNFSM5RZ63D4A . You are receiving this because you commented.Message ID: @.***>

[ColinKennedy]

@fnaum have you had a chance to check this? Just curious if the code exists and you / AL would be willing to PR it back into Rez or if we'll want to target this as a separate effort.

[fnaum]

Hey @colin I haven't but I will next week. Definitely willing to PR it back, just need to find the time ;) F

On Thu, Apr 14, 2022 at 7:21 AM Colin Kennedy @.***> wrote:

@fnaum https://github.com/fnaum have you had a chance to check this? Just curious if the code exists and you / AL would be willing to PR it back into Rez or if we'll want to target this as a separate effort.

— Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1263#issuecomment-1098498055, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVRQRD3W34PQSOJ2EF2HKTVE43GNANCNFSM5RZ63D4A . You are receiving this because you were mentioned.Message ID: @.***>

[hughetop]

This sounds similar to what we do, but perhaps more complex. I already saw someone suggest saving the resolve to a .rxt file and then loading that to get exactly the same packages and versions.

"If your example of -d here means to say, "add this path to REZ_PACKAGES_PATH instead of me having to change my rezconfig", then that's already very similar to the existing --paths option. I'll grant you it is a bit more user friendly though (since you don't have to repeat the rest of the packages path), so if that is what you mean, and if you named it something like --add-pre-paths/--add-post-paths instead, then I wouldn't be opposed to adding that."

A trick we do is inside a global .cshrc that everyone uses, we define export REZ_PACKAGES_PATH=~/packages:/some/network/path/int:/some/network/path/ext which is exactly the same as the packages listed in rezconfig.py, but the benefit is we have that available at runtime. So we can do rez-env --paths=/some/network/test/area:$REZ_PACKAGES_PATH helloworld-1.2.3 in order to test packages outside of the normal paths defined in rezconfig. You want to specify the package version so it doesn't accidentally pick up newer builds. This avoids the homedir, allows other users (or farm jobs) to test your changes, but doesn't affect any official released packages or the rezzconfig.

[ColinKennedy]

That's pretty much what we do as well, it just isn't ideal. As mentioned in previous posts, a package orderer would solve the problem of "what if helloworld 1.2.3.1 gets released outside of /some/network/test/area". You could modify the request to helloworld==1.2.3 to force it to use yours, but then that requires the request to change.

Suggested interface:

rez-env --favor-paths If specified but no paths provided, use [config.local_packages_path]

rez-env --favor-paths /foo:/bar:/thing If specified with paths, all Rez package families / versions found in those parts will be preferred over anything else found in config.packages_path.

[ColinKennedy]

@nerdvegas I gave this a shot and got it mostly working. There's a WIP located over at

https://github.com/nerdvegas/rez/compare/master...ColinKennedy:issues/1263-add_favor_paths_package_orderer?expand=1

If you have two paths

/foo
    package_a/
        1.0.0/
/bar
    package_a/
        2.0.0/

REZ_PACKAGES_PATH=/bar:/foo rez-env package_a --favor-paths /foo resolves to 1.0.0 from /foo as expected. However this doesn't 100% solve the problems I'd mentioned in previous posts in this thread.

/foo
    package_a/
        1.0.0/
/bar
    package_a/
        1.0.0/

REZ_PACKAGES_PATH=/bar:/foo rez-env package_a --favor-paths /foo resolves to 1.0.0 from /bar instead of /foo. It seems the reason is related to if key in seen:, from iter_packages.

Since /foo and /bar both have (package_a.name, package_a.version), the iterable which gets passed to FavorPathsOrder.reorder completely omits the package_a 1.0.0 from /foo. So in short, the sort logic works but we cannot sort if the packages aren't ever passed to FavorPathsOrder.reorder. An exact-match package versions still have the potential to resolve from /bar instead of the favored /foo.

The test_conflicting_versions unittest demonstrates the issue. The test should pass, but fails instead, pointing to the "favor_paths_first" folder.

I'm happy to PR this edit in since it is an improvement over what we currently have. However for the exact-match case, the only solution I can think of is to make a new, unique version for package_a in /foo that I know cannot exist in /bar. Like just rez-build a version called 1.0.0-asdf or anything. It's not a great solution. Is there something cleaner that we can do? @fnaum tagging you as well in case you know of a solution.

[halil3d]

@ColinKennedy I'm curious if /foo happened to also be the local_packages_path, would this then work as expected? While this FavorPathsOrder functionality would be nice to allow any path to be favoured, I think if only the local path was favoured for older package versions, this would solve quite a few use cases that have been discussed. I'm not familiar with the specific Animal Logic implementation, but I believe it is specific to the local_packages_path?

[ColinKennedy]

I'd have to double check when I'm on a machine with Rez but I think I tried matching REZ_LOCAL_PACKAGES_PATH and --favor-paths to the same path and it was the same (unwanted) result. Because of the iter_packages behavior as mentioned.

While this FavorPathsOrder functionality would be nice to allow any path to be favoured, I think if only the local path was favoured for older package versions, this would solve quite a few use cases that have been discussed.

Sorry, could you possibly rephrase? I'm not sure I follow. The behavior currently is

1. rez-env --favor-paths <-- favors local_packages_path
2. rez-env --favor-paths /blah <-- favors /blah

So you could get the "only favor local_packages_path" behavior by simply not providing a path after --favor-paths, already.

Again, I might've misunderstood. But from your reply, it sounded like the behavior you're looking for is already working. It's been a while since I worked at Animal Logic but I believe theirs only provides functionality (1), where we also can do functionality (2).

[ColinKennedy]

I made a PR with the suggested edits. Btw to any interested parties - I was thinking we should probably also favor paths as a configuration option (not just as a CLI flag). Would that be alright to add? In this PR or a second one).

The logic would follow as - if provided as CLI flag, always use that. Otherwise use the rez.config.config option, assuming that it's set.

[halil3d]

So you could get the "only favor local_packages_path" behavior by simply not providing a path after --favor-paths, already.

Again, I might've misunderstood. But from your reply, it sounded like the behavior you're looking for is already working. It's been a while since I worked at Animal Logic but I believe theirs only provides functionality (1), where we also can do functionality (2).

Ah perfect, I didn't catch that the default behaviour provided that outcome if no args were passed (1). That is exactly what I was looking to confirm was possible. Hopefully there's a way around the caveat you mention for (2). Thanks for putting up the PR.

[fnaum]

Hey Colin,

I think AL implementation works fine for the case you depicted that is not working for you, but as Halil guessed it is tied to the REZ_LOCAL_PACKAGES_PATH, we do not have the option to pass a path to be preferred, but of course, we can change the REZ_LOCAL_PACKAGES_PATH to whatever we need, and that is what we do in the CI, we change that to a temp folder so all the packages that are built as part of a test are preferred or enforced to come from there.

We are finalizing the CLA "paperwork" for the contributions. I hope that once that is ready we can push that and you can compare it or use it.

Cheers, Fede

On Mon, May 9, 2022 at 10:07 AM halil3d @.***> wrote:

So you could get the "only favor local_packages_path" behavior by simply not providing a path after --favor-paths, already.

Again, I might've misunderstood. But from your reply, it sounded like the behavior you're looking for is already working. It's been a while since I worked at Animal Logic but I believe theirs only provides functionality (1), where we also can do functionality (2).

Ah perfect, I didn't catch that the default behaviour provided that outcome if no args were passed (1). That is exactly what I was looking to confirm was possible. Hopefully there's a way around caveat you mention for (2). Thanks for putting up the PR.

— Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1263#issuecomment-1120514215, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVRQRBLMYWMOUJ7AUI2CTTVJBJNJANCNFSM5RZ63D4A . You are receiving this because you were mentioned.Message ID: @.***>

[ColinKennedy]

@fnaum the PR should over at https://github.com/AcademySoftwareFoundation/rez/pull/1305 should have everything. Not sure how to get the PR merged but it's fully tested and ready for whenever anyone wants to review it.