ocaml
Virtual packages are not supported on Suse
opam install conf-python-3 fails because for some reason rpm -q --whatprovides python3 == python313-3.13.2-3.1.x86_64 is not recognized.
Is there some sort of --debug-solver knob to see what attempts opam 2.3.0 actually does to find something that provides python3?
This is a regression between 2.0.10 and 2.1.0~beta2:
opam init
opam install opam-devel.2.0.10
alias opam2="OPAMROOT=~/.opam2 ~/.opam/default/lib/opam-devel/opam"
rm -rf ~/.opam2 ; opam2 --version && opam2 init --disable-sandboxing && opam2 install conf-python-3
opam install opam-devel.2.1.0~beta2
rm -rf ~/.opam2 ; opam2 --version && opam2 init --disable-sandboxing && opam2 install conf-python-3
opam 2.2.0 works just fine on Fedora. I believe that issue was fixed in https://github.com/ocaml/opam/pull/4791
And to answer your question, the way to debug the depexts system is to use either -vvvv or set OPAMVERBOSE=4. For example:
opam install --dry-run -vvvv conf-python-3
Thanks for looking into this. c76a512a4d887a6e9623c6f5447ff8d26104cc49 is in 2.2.0, and does apparently not fix the bug.
Maybe it is broken just in Tumbleweed, Leap still has a package python3:
zypper --no-refresh search --provides --type package --match-exact --verbose --details python3
S | Name | Type | Version | Arch | Repository
---+-----------+---------+------------+--------+-----------
i | python313 | package | 3.13.2-3.1 | x86_64 | rmt-oss
provides: python3 = 3.13
vs.
i+ | python3 | package | 3.6.15-150300.10.78.1 | x86_64 | SLE-Module-Basesystem15-SP6-Updates
provides: python3 = 3.6
provides: python3 = 3.6.15-150300.10.78.1
I see a call zypper "--quiet" "se" "-t" "package", which does not list python3 as a package name.
Perhaps it need to be clarified what ["python3"] actually means: is it a plain rpm package name, or is it some "virtual" thing, something that can be either a plain rpm package name, or something that could also come via a Provides:.
ah, given your ticket description and title i wrongly assumed your system was Fedora adjacent instead of Suse. Next time please do provide your system info using opam config report as mentioned in our issue template.
I'll update the title
There is also a more general ticket about our support of virtual system packages in https://github.com/ocaml/opam/issues/4759
Looking through packages_status, it seems that function works with plain package names, by requesting a list via zypper. There is no easy way to get a list of all Provides:, the output of zypper --no-refresh search --provides --type package --details --verbose is complicated to parse.
Maybe the approach is unsuitable: If some_name is requested, just do zypper install --dry-run some_name and see if it resolves to some real rpm package. An alternative would be to link to libzypp.so and get the data from there.
I only speak gawk. Someone needs to translate this to Re to cover every provided string, not just plain rpm package names. The regex does also cover pkgconfig(something), because this is what dune-configurator will search for. But once such provides are used in opam files, it means they are only compatible starting with opam 2.3.1.
#!/bin/bash
unset LANG
unset ${!LC_*}
output_i=~/prov_i.txt
output_n=~/prov_n.txt
test -f "${output_i}" || zypper --no-refresh search --installed-only --details --verbose --type package > "${output_i}"
test -f "${output_n}" || zypper --no-refresh search --not-installed-only --details --verbose --type package > "${output_n}"
gawk '
/^[[:blank:]]*i[[:blank:]]+\|[[:blank:]]+[^[:blank:]]+[[:blank:]]+\|[[:blank:]]+package[[:blank:]]+\|/{print $3}
/^[[:blank:]]+provides:[[:blank:]]+(pkgconfig\([^)]+\)|[-_\.a-zA-Z0-9]+)($|[[:blank:]])/{print $2}
' "${output_i}" | sort -u | less -Sn
gawk '
/^[[:blank:]]+\|[[:blank:]]+[^[:blank:]]+[[:blank:]]+\|[[:blank:]]+package[[:blank:]]+\|/{print $2}
/^[[:blank:]]+provides:[[:blank:]]+(pkgconfig\([^)]+\)|[-_\.a-zA-Z0-9]+)($|[[:blank:]])/{print $2}
' "${output_n}" | sort -u | less -Sn
This simplified version of the gawk commands above does not compile, but should give an idea how it could be done:
--- a/src/state/opamSysInteract.ml
+++ b/src/state/opamSysInteract.ml
@@ -507,16 +507,13 @@ let packages_status ?(env=OpamVariable.M
Re.(Group.get (exec re_pkg l) 1) +++ pkgs
with Not_found -> pkgs) OpamSysPkg.Set.empty
in
- let with_regexp_dbl ~re_installed ~re_pkg =
- List.fold_left (fun (inst,avail) l ->
+ let with_regexp_dbl ~re_pkg =
+ List.fold_left (fun output l ->
try
let pkg = Re.(Group.get (exec re_pkg l) 1) in
- if Re.execp re_installed l then
- pkg +++ inst, avail
- else
- inst, pkg +++ avail
- with Not_found -> inst, avail)
- OpamSysPkg.Set.(empty, empty)
+ pkg +++ output
+ with Not_found -> output)
+ OpamSysPkg.Set.empty
in
let package_set_of_pkgpath l =
List.fold_left (fun set pkg ->
@@ -978,28 +975,69 @@ let packages_status ?(env=OpamVariable.M
compute_sets sys_installed
| Suse ->
(* get the second column of the table:
- zypper --quiet se -i -t package|grep '^i '|awk -F'|' '{print $2}'|xargs echo
- output:
+ zypper --quiet --no-refresh search --details --verbose --type package
+ zypper --quiet --no-refresh search --details --verbose --type package
+ output with --installed-only:
>S | Name | Summary
>--+-----------------------------+-------------
- > | go-gosqlite | Trivial SQLi
>i | libqt4-sql-sqlite-32bit | Qt 4 sqlite
+ > provides: libqt4-sql-sqlite-32bit = version
+
+ output with --not-installed-only:
+ >S | Name | Summary
+ >--+-----------------------------+-------------
+ > | go-gosqlite | Trivial SQLi
+ > provides: go-gosqlite = version
*)
+ let sys_installed =
let re_pkg =
Re.(compile @@ seq
[ bol;
- rep1 any;
+ rep space;
+ char 'i';
+ rep1 space;
char '|';
rep1 space;
group @@ rep1 @@ alt [alnum; punct];
rep1 space;
char '|';
+ ],
+ [ bol;
+ rep1 space;
+ char '|';
+ rep1 space;
+ str "provides:";
+ rep1 space;
+ group @@ rep1 @@ alt [alnum; punct];
+ alt [eol; space];
])
in
- let re_installed = Re.(compile @@ seq [bol ; char 'i']) in
- let sys_installed, sys_available =
- run_query_command "zypper" ["--quiet"; "se"; "-t"; "package"]
- |> with_regexp_dbl ~re_installed ~re_pkg
+ run_query_command "zypper" ["-q"; "--no-refresh"; "se"; "--provides"; "-v"; "-s"; "-t"; "package"; "-i"]
+ |> with_regexp_dbl ~re_pkg
+ in
+ let sys_available =
+ let re_pkg =
+ Re.(compile @@ seq
+ [ bol;
+ rep space;
+ char '|';
+ rep1 space;
+ group @@ rep1 @@ alt [alnum; punct];
+ rep1 space;
+ char '|';
+ ],
+ [ bol;
+ rep1 space;
+ char '|';
+ rep1 space;
+ str "provides:";
+ rep1 space;
+ group @@ rep1 @@ alt [alnum; punct];
+ alt [eol; space];
+ ])
+ in
+ run_query_command "zypper" ["-q"; "--no-refresh"; "se"; "--provides"; "-v"; "-s"; "-t"; "package"; "-u"]
+ |> with_regexp_dbl ~re_pkg
in
compute_sets sys_installed ~sys_available
Using this zypper command makes it so that a large number of opam commands now take an extra 10s (on a fast laptop).
At the moment i'm thinking more of taking the route of using rpm directly instead and using the case we already have for Centos and Altlinux (which does not provide support for package availability)
rpm -qa --qf '%{NAME}\n%{PROVIDES}\n' takes less than a second on my system, lines with braces and slashes should be removed from the output.
zypper --quiet --no-refresh search --type package --verbose --not-installed-only takes 10 seconds. I will see if there is a way to improve the query for available packages.
Is there an easy way to run two commands in parallel?
Edit: needs brackets: rpm -qa --qf '%{NAME}\n[%{PROVIDES}\n]'
Edit: zypper --quiet --no-refresh search --type package --verbose --not-installed-only takes 9 seconds,
zypper --quiet --no-refresh search --type package --verbose --provides --not-installed-only '*' takes 4 seconds.
Thanks for the pointer on %{PROVIDES}. I've opened a PR in https://github.com/ocaml/opam/pull/6431. There are some downsides to that fix but overall it's probably the only thing we can do at the moment.
What functionality is lost by removing support for "available packages"?
It seems the behavior changes from an incomplete list with zypper --quiet search --type package, which takes ~3.45s, to a complete list with rpm -qa --qf '%{NAME}\n%{PROVIDES}\n', which takes ~0.45s.
In case it is desirable to have "available packages": zypper --quiet --no-refresh search --type package --verbose --provides --not-installed-only '*', which would take ~4.30s.
What functionality is lost by removing support for "available packages"?
It would disable the feature which make opam packages automatically unavailable (as if they had available: false) when a required depexts package is also unavailable. For example this allows to automatically choose the right version of the conf-llvm package without the high maintenance cost (for opam-repository maintainers and contributors) associated with recording which versions are available on every single distribution and distribution version (not just distribution families).
However outside of these kind of complicated packages (e.g. conf-llvm), the utility of the feature is more debatable.
which would take ~4.30s.
4s is way too long for this. Package availability is calculated for a number of opam commands (e.g. opam show, opam install, …) so this would add 4s to the time this command takes.
The fact that the current command on SUSE-based systems takes 3s is a bug due to a lack of testing.
I applied 5ffd07705853a7d6516b65db08d3f0a47aa421ab + d5e0bcb805839ce8e9b9e423415bd81fc8c13abe + 2532248d70b14a137164194e6c0eb3459b50de0f on e13109411952d4f723a165c2a24b8c03c4945041.
Now rm -rf ~/.opam&&opam init --disable-sandboxing --compiler=5.1.1&&opam install coccinelle.1.3.0 works. Thanks again.
In case some rpm package is missing, zypper will install it. There is some weird formatting with an interactive zypper. But I have seen this occasionally elsewhere. It depends on the used console and the TERM variable. When I find the time I will see what can be done about it.