conan-io
[question] Ignore local cache when creating lock file
What is your question?
I see that conan lock create has option --no-remote which prevents the use of the remote and resolves exclusively in the cache. But is there a way to exclusively use remotes, without a local cache?
I see that even by defining specific remotes the local cache is consulted. This is problematic as a developer who is working on updating the lock file may have local packages that don't exist in the remote and then the lock file is generated with respect to them. When such a lock file is then used on the CI or by other developers, conan fails because it can't resolve locked recipe IDs.
Note: I'm using lock API from a custom command using the code below, but I think the question still remains the same:
remotes = conan_api.remotes.list()
graph = conan_api.graph.load_graph_consumer(
path=conanfile_path,
name=None,
version=None,
user=None,
channel=None,
profile_host=profile_host,
profile_build=profile_build,
lockfile=None, # don't use old lockfile as basis
remotes=remotes,
update=True,
)
if parsed_args.verbose:
print_graph_basic(graph)
graph.report_graph_error()
conan_api.graph.analyze_binaries(
graph,
None,
remotes=remotes,
)
if parsed_args.verbose:
print_graph_packages(graph)
lockfile = conan_api.lockfile.update_lockfile(
lockfile=None, # don't use old lockfile as basis
graph=graph,
# only recipe should be locked, not the package ID, which depends on arch, os, and build type
lock_packages=False,
clean=True,
)
Have you read the CONTRIBUTING guide?
- [ ] I've read the CONTRIBUTING guide
To reproduce the issue, you can create a simple package that build-requires nasm/2.14 from conan-center.
Then, simply download nasm/2.14#d4c0f102da173fe6ea0fe5c486a29709 from conancenter and upload it to your local artifactory instance.
Then, run conan install --tool-requires nasm/2.14 to make sure that the newer revision from conancenter is fetched and installed to your local cache (this is currently nasm/2.14#255a93a9bcacb93baab9543165a484ea). After that, remove conancenter from your list of remotes with conan remote remove conancenter and run conan lock create . -r myremote.
You will see that the lockfile contains 255a93a9bcacb93baab9543165a484ea revision of nasm/2.14, instead the latest from private remote, namely d4c0f102da173fe6ea0fe5c486a29709, even though conancenter is no longer in the list of remotes - the 255a93a9bcacb93baab9543165a484ea just exists in your local cache. This is the same as if the developer created a local revision of some package which is then used for locking instead of the ones available from the remotes.
I'm currently using conan 2.0.10.
This is problematic as a developer who is working on updating the lock file may have local packages that don't exist in the remote and then the lock file is generated with respect to them. When such a lock file is then used on the CI or by other developers, conan fails because it can't resolve locked recipe IDs.
+1
I agree, it would be nice to have an option like --no-cache added to conan lock create.
Currently using conan v2.2.2, and the only workaround I found is to delete the packages from the cache that are listed in the conan.lock (conan remove --confirm <package>), but it's annoying and error prone.
I am afraid that this is not possible.
The conan dependency graph is evaluated from the cache. The lockfile is just a snapshot of a dependency graph, and that is not expected to change. Whatever is in a dependency graph, that is what is stored in a lockfile, no more no less. A lockfile itself cannot lock something different, it is like a picture of a scene, it cannot capture something different that the reality, that for it it will be the current dependency graph.
Now, this means that you would be asking for a way to compute a dependency graph guaranteeing that all dependencies do exist in the server. But this is also not something possible, lets say that you have a conanfile that depends on something like mypkg/0.1, and it finds a recipe revision of it in the Conan cache, which is revision_abc. Let's say that the algorithm can check if this specific revision exist in the servers and then you find that it doesn't, and in the servers only revision_xyz exists, which is older than the current cache revision_abc. Should the current cache one be evicted (removed) from the cache? Certainly not expected, this sounds like a revision that the developer put there. But if the server revision is older, it is the only way, as revisions order cannot be forcely altered to change their order, because that causes "instability" in the behavior.
Note that this is not optional, the revision from the server must be downloaded to be able to be evaluated and the graph expanded, it is not possible to evaluate things just on memory, they need to be brought to the cache. So it must be given a local recipe revision with the server timestamp, and unless this is previously locked (if it is previously locked, then the whole of this is not an issue, it already works), then it has to be the latest revision in the cache, otherwise it cannot be resolved, but this cannot happen if there is another recipe revision more modern in the cache.
Currently using conan v2.2.2, and the only workaround I found is to delete the packages from the cache that are listed in the conan.lock (conan remove --confirm
), but it's annoying and error prone.
I don't fully get this. If the packages are already listed in the lockfile, the revisions are already locked there. You mean removing them and then capturing the lockfile again?
Probably I am not fully understanding the flow here, but if there is a newer revision in the cache, it is because it was put there on purpose, and if it is more modern, the intent would be to use it. If that is not the goal, why not working with the lockfile of the dependencies that were uploaded to the server in the previous state as a base?
Probably I am not fully understanding the flow here, but if there is a newer revision in the cache, it is because it was put there on purpose, and if it is more modern, the intent would be to use it.
Some packages cannot be made editable-friendly (and lot's of packages from conan-center as such as well) because they have complex package_info logic that does not work in the editable mode. So, instead of using the editable mode, a common case is to simply export the package to the local cache and develop against it. Sometimes, however, you need to work on multiple different repositories and then the exported package for the purpose of development of repo1 can interfere with lockfile generation of repo2 which is not yet ready to use the new package. When such lockfile is then committed and used on different machine, conan installation fails because the locked version exists only locally on the machine of the developer that made the lockfile.
I agree, it would be nice to have an option like --no-cache added to conan lock create.
Actually, this gave me an idea. You can achieve something like this with env CONAN_HOME=/tmp/folder conan lock create .. It will temporarily use /tmp/folder as conan home folder which will not contain anything in cache and thus ensure that no local packages accidentally end-up in lockfile.
Actually, this gave me an idea. You can achieve something like this with env CONAN_HOME=/tmp/folder conan lock create .. It will temporarily use /tmp/folder as conan home folder which will not contain anything in cache and thus ensure that no local packages accidentally end-up in lockfile.
Recall that different projects can have a .conanrc file that defines a new CONAN_HOME folder for it automatically, so different projects can have different caches so they don't interfere with each other, if that was an issue.
Recall that different projects can have a .conanrc file that defines a new CONAN_HOME folder for it automatically
Cool! I wasn't aware of that.
I am still transitioning from conan 1 without lockfile to conan 2 with lockfile, so sorry if I'm not fully clear with conan 2 lockfiles design.
Now, this means that you would be asking for a way to compute a dependency graph guaranteeing that all dependencies do exist in the server.
Exactly. To me, the server is the groundtruth (where packages are uploaded from the CI), and my local cache is a (dirty) mix of packages from the server and packages I built locally (and that are not available to others developers). So when I'm creating a lockfile (from scratch) or updating an existing one, it can happen that it points to a package only available in my local cache. If that's the case, when I push the lockfile to origin, the CI will try to build the package using the lockfile but it will fail (because it cannot find the package).
But this is also not something possible, lets say that you have a conanfile that depends on something like mypkg/0.1, and it finds a recipe revision of it in the Conan cache, which is revision_abc. Let's say that the algorithm can check if this specific revision exist in the servers and then you find that it doesn't, and in the servers only revision_xyz exists, which is older than the current cache revision_abc. Should the current cache one be evicted (removed) from the cache? Certainly not expected, this sounds like a revision that the developer put there.
That's why in this example, I think an option conan lock create . --no-cache could make sense. As a developer, if I add the --no-cache, then the packages from the current cache should be ignored, and it's the behavior I expect (because --no-cache is not a default).
I don't fully get this. If the packages are already listed in the lockfile, the revisions are already locked there. You mean removing them and then capturing the lockfile again?
Probably I am not fully understanding the flow here, but if there is a newer revision in the cache, it is because it was put there on purpose, and if it is more modern, the intent would be to use it. If that is not the goal, why not working with the lockfile of the dependencies that were uploaded to the server in the previous state as a base?
In my case, this was the scenario:
conancenterremote disabled- private remote enable (let's call it
jfrog) - package A without lockfile (for now), that depends on package B
I work on package B (calling conan package many times), I am happy with the recipe, I push to origin, create a PR, merge and release. The CI will build and upload the package B to jfrog. My local cache is full of dirty package B.
I then go to package A, run conan lock create . --remote jfrog, but the lockfile points to a revision of package B in my cache.
Actually, this gave me an idea. You can achieve something like this with env CONAN_HOME=/tmp/folder conan lock create .
It does not work if you are working with a private remote defined in your ~/.conan2/remotes.json
It does not work if you are working with a private remote defined in your ~/.conan2/remotes.json
You can do the same conan config install you do in your main Conan home, in every project/.conanrc home that you want. Or conan remote add. It is an independent cache, with its own remotes definition.
Adding to this issue, it would also be really useful to have a --no-cache option when creating package lists (I do this with conan install). I've just run into an issue where I was unable to promote using a package list because packages happened to be in the cache, instead of downloaded from the remote they were supposed to come from.
At present, my only option is to clear the cache (this is okay, but could become needlessly time consuming if a user ran the promotion script locally and lost their cache), or try the CONAN_HOME hack proposed above.
Note that this is not optional, the revision from the server must be downloaded to be able to be evaluated and the graph expanded, it is not possible to evaluate things just on memory, they need to be brought to the cache.
I understand this, but if the --no-cache option were used, couldn't a temporary cache be used, which would be deleted once the evaluation is done? A user specifying --no-cache should understand that is will be less efficient than using the cache.
FYI @antoinebardoz
In the end, I ended up creating a custom command in ~/.conan2/extensions/commands/org/cmd_lock.py distributed to my team with conan config install https://github.com/Org/conan_config.git
It can be called with conan org:lock update .
Summary:
- move the
~/.conan2/p/in a temp directory under/tmp/tmp.xxx/ - do the same as a
conan lock update .(which will download the revisions from the server in the new cache) - emit a warning if there are editable packages (which can mess up the created
conan.lock) - delete
~/.conan2/p/and move back the previous cache - if there is an error, put everything back as it was before
Source code:
import argparse
import os
import tempfile
import shutil
from pathlib import Path
from typing import List
from conan.api.conan_api import ConanAPI
from conan.api.output import ConanOutput
from conans.errors import ConanException
from conan.cli.args import common_graph_args, validate_common_graph_args
from conan.cli.command import conan_command, conan_subcommand, ConanArgumentParser
from conan.cli.printers.graph import print_graph_packages, print_graph_basic
from conans.model.graph_lock import LOCKFILE
from conans.client.graph.graph import DepsGraph, Node
@conan_command(group="Organization commands")
def lock(conan_api: ConanAPI, parser, *args):
"""
Extend Conan lock command.
"""
@conan_subcommand()
def lock_update(
conan_api: ConanAPI,
parser: ConanArgumentParser,
subparser: ConanArgumentParser,
*args,
):
"""
Update a lockfile from a conanfile or a reference, without using the local cache.
See https://github.com/conan-io/conan/issues/14613 for more details.
"""
common_graph_args(subparser)
_args = parser.parse_args(*args)
validate_common_graph_args(_args)
_lock_update(conan_api, _args)
def _get_editable_packages(graph: DepsGraph) -> List[Node]:
editable = []
for node in graph.nodes:
if node.recipe == "Editable":
editable.append(node)
return editable
def _conan_lock_create(conan_api: ConanAPI, args: argparse.Namespace):
cwd = os.getcwd()
path = conan_api.local.get_conanfile_path(args.path, cwd, py=None)
remotes = conan_api.remotes.list(args.remote) if not args.no_remote else []
overrides = eval(args.lockfile_overrides) if args.lockfile_overrides else None
lockfile = conan_api.lockfile.get_lockfile(
lockfile=args.lockfile,
conanfile_path=path,
cwd=cwd,
partial=True,
overrides=overrides,
)
profile_host, profile_build = conan_api.profiles.get_profiles_from_args(args)
graph = conan_api.graph.load_graph_consumer(
path,
args.name,
args.version,
args.user,
args.channel,
profile_host,
profile_build,
lockfile,
remotes,
args.update,
is_build_require=False,
)
print_graph_basic(graph)
graph.report_graph_error()
conan_api.graph.analyze_binaries(
graph, args.build, remotes=remotes, update=args.update, lockfile=lockfile
)
print_graph_packages(graph)
lockfile = conan_api.lockfile.update_lockfile(
lockfile, graph, args.lockfile_packages, clean=args.lockfile_clean
)
conanfile_path = (
os.path.dirname(graph.root.path)
if graph.root.path and args.lockfile_out is None
else cwd
)
conan_api.lockfile.save_lockfile(
lockfile, args.lockfile_out or LOCKFILE, conanfile_path
)
editables = _get_editable_packages(graph)
if editables:
out = ConanOutput()
out.warning("The following packages are editable:")
for node in editables:
out.warning(f" {node.ref}")
out.warning("Editable packages cannot be locked, so it might cause issues.")
out.warning("All your editable packages can be removed with:")
out.warning(' conan editable remove -r "*"')
def _lock_update(conan_api: ConanAPI, args: argparse.Namespace):
if not args.path:
raise ConanException("Please provide a path")
home = Path(conan_api.config.home())
home_p = home / "p"
if not home_p.is_dir():
raise ConanException(f"{home_p} does not exist.")
temp_home = Path(tempfile.mkdtemp())
temp_home_p = temp_home / "p"
out = ConanOutput()
out.writeln(f"Temporary folder {temp_home}")
shutil.move(str(home_p), str(temp_home))
conan_dir = Path(args.path).absolute()
lockfile = conan_dir / LOCKFILE
temp_lockfile = temp_home / LOCKFILE
if lockfile.is_file():
shutil.move(str(lockfile), str(temp_lockfile))
try:
_conan_lock_create(conan_api, args)
except ConanException as e:
if not lockfile.is_file():
# if something goes wrong, make sure to keep the old lockfile
shutil.move(str(temp_lockfile), str(lockfile))
raise e
finally:
if home_p.is_dir():
# move the new cache (from the lock create command) out of the way
shutil.move(str(home_p), str(temp_home / "p_remote"))
# restore the old cache back
shutil.move(str(temp_home_p), str(home))
But yes, an option --no-cache would be appreciated.
@nathan-b-flya , I like your solution. I will steal it and put it in my own lock wrapper custom command 😛