Improve conda setup

[tobiasdiez]

A few small quality of life improvements to the conda setup. Notably the environment files under src are now moved to the root, as the old env files there were non-functional and only confused users. Relately, removed the outdated,not working and untested instructions to use conda solely to provide the system packages for sage-the-distro.

A few other improvements along the way:

• Make conda in devcontainer working again by forcing mamba v1 (v2 was released a few days ago and breaks a few things related to the lock files)
• Force usage of conda-forge everywhere (mixing of channels is no longer supported)

:memo: Checklist

• [ ] The title is concise and informative.
• [ ] The description explains in detail what this PR is about.
• [ ] I have linked a relevant issue or discussion.
• [ ] I have created tests covering the changes.
• [ ] I have updated the documentation and checked the documentation preview.

:hourglass: Dependencies

[github-actions[bot]]

Documentation preview for this PR (built with commit cf31bde9bfd6b788f9956781b602034ff292e995; changes) is ready! :tada: This preview will update shortly after each push to this PR.

[kwankyu]

Is it not possible to put all the environment files into some directory, like .environment? They make the sage root directory look cluttered...

[tobiasdiez]

Is it not possible to put all the environment files into some directory, like .environment? They make the sage root directory look cluttered...

Conda-lock has the possibility to create "multi-platform" lock files. That would reduce the number of lock files. I can look into this afterwards.

[kwankyu]

Nice. Thanks.

[kwankyu]

How am I supposed to use the default devcontainer using conda?

(base) @kwankyu ➜ /workspaces/sage (develop) $ ./sage
Traceback (most recent call last):
  File "/workspaces/sage/src/bin/sage-ipython", line 9, in <module>
    from sage.misc.banner import banner
ModuleNotFoundError: No module named 'sage'

(base) @kwankyu ➜ /workspaces/sage (develop) $ conda activate sage-dev
(sage-dev) @kwankyu ➜ /workspaces/sage (develop) $ sage
bash: sage: command not found
(sage-dev) @kwankyu ➜ /workspaces/sage (develop) $ ./sage
Traceback (most recent call last):
  File "/workspaces/sage/src/bin/sage-ipython", line 9, in <module>
    from sage.misc.banner import banner
ModuleNotFoundError: No module named 'sage'

[kwankyu]

Actually it seems that codespaces failed in the setup stage:

[tobiasdiez]

Thanks for testing. I can reproduce the issue. It fails with:

Building wheels for collected packages: sagemath-standard
  Created temporary directory: /tmp/pip-wheel-zedyx3_w
  Destination directory: /tmp/pip-wheel-zedyx3_w
  Building editable for sagemath-standard (pyproject.toml): started
  Building editable for sagemath-standard (pyproject.toml): finished with status 'error'
Failed to build sagemath-standard
Exception information:
Traceback (most recent call last):
  File "/opt/conda/envs/sage-dev/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/opt/conda/envs/sage-dev/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/sage-dev/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 429, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: Could not build wheels for sagemath-standard, which is required to install pyproject.toml-based projects

Not very informative sadly. Does it print more for you?

For me it works when I call pip install --no-build-isolation -v -v -e ./src manually in the terminal (after activating sage-dev). Could you give this a try?

[kwankyu]

As you see in the screenshot, the devcontainer seems to fail somehow before even getting to building sagemath-standard...

[tobiasdiez]

try to start the codespace with more space: 2-core • 8GB RAM • 32GB • 14.11 GB works for me

[kwankyu]

try to start the codespace with more space: 2-core • 8GB RAM • 32GB • 14.11 GB works for me

2-core • 8GB RAM • 32GB is the default codespaces machine type for me. I don't know what that 14.11 GB is.

Trying again.

[kwankyu]

It failed exactly at the same step, as shown in the above screenshot.

[tobiasdiez]

Try to use https://docs.github.com/en/codespaces/customizing-your-codespace/changing-the-machine-type-for-your-codespace#changing-the-machine-type with 64GB. The 14GB is the amount of space it uses for me and you can see this at https://github.com/codespaces.

[kwankyu]

It seems that there is no machine type with 64GB disk space available for me. I am on free tier.

You are using 14GB. The default machine already has 32GB. Hence the disk space should not be a problem...

[tobiasdiez]

I agree, it's strange that you run out of space. In the background of the screenshot you shared, you also see that the codespace actually was created successful - just afterwards it runs out of space.

But maybe we leave this for another PR? The only change I made to the codespace setup was to fix the conda initialization, which is also working for you (sage-dev is correctly initialized in your codespace, just sage failed to build).

[kwankyu]

I agree, it's strange that you run out of space. In the background of the screenshot you shared, you also see that the codespace actually was created successful - just afterwards it runs out of space.

It seems really a space problem. There indeed appeared log messages (strangely above the "success" message) like:

...
  [Errno 28] No space left on device: '/workspaces/sage/src/sage/rings/polynomial/polynomial_rational_flint.cpp'
...

But maybe we leave this for another PR? The only change I made to the codespace setup was to fix the conda initialization, which is also working for you (sage-dev is correctly initialized in your codespace, just sage failed to build).

Are you using a bigger machine, say with 64 GB disk space?

It is really problematic if the default 32 GB machine could not cope with the conda install, since this is the default devcontainer.json...

[dimpase]

by the way, why is there no environment-3.12-linux.yml ?

[kwankyu]

I will leave review to Dima.

[tobiasdiez]

It is really problematic if the default 32 GB machine could not cope with the conda install, since this is the default devcontainer.json...

I agree. But the conda dependencies only take 4-5GB or so. Not sure how much the OS in the codespace takes, but shouldn't be more than 5GB. So the problematic part is that in order to build sage one needs >= 10-15GB. This is also about the same size where we hit issues with the build github actions a year ago...so not really a new problem.

by the way, why is there no environment-3.12-linux.yml ?

Will be added in https://github.com/sagemath/sage/pull/36431.

I've checked on a local Linux conda install with Python 3.11, running ... the resulting ./sage seems to work.

Thanks for testing!

[kwankyu]

...This is also about the same size where we hit issues with the build github actions a year ago...so not really a new problem.

Perhaps not new.

If the conda install does not work with the default 32 GB machine, we should not set it as the default devcontainer for developers who want to use codespaces for sage development. The PR #38789 adds a devcontainer that is ready to build sage from source. We may need to consider to change the default devcontainer to it...

[tobiasdiez]

I'm setting this now to positive review based on the positive review and feedback by @dimpase and @vincentmacri (thanks!), since this is urgently needed to fix the ci-issues that will be introduced by https://github.com/sagemath/sage/pull/38804.

We can always readd the outdated instructions that I've removed from the docs in a later PR.

[dimpase]

fine with me to give it a positive review

[tobiasdiez]

the merge was automatic, thus setting back to positive review.

[vincentmacri]

@tobiasdiez I'm not able to get a non-editable conda install working. It builds but some basic functionality is broken. For instance:

P2.<X, Y, Z> = ProjectiveSpace(2, QQ)
C = Curve(X^3 + Y^3 + Z^3)

is giving me

AttributeError: 'sage.rings.polynomial.multi_polynomial_libsingular.MPolynomialRing_libsingular' object has no attribute '_CategoryObject__gens_dict'

whereas I should be able to print(C) and get Projective Plane Curve over Rational Field defined by X^3 + Y^3 + Z^3.

Can you confirm whether the non-editable conda install works for you? If it does, can you add to the documentation the steps to do a non-editable install? If it doesn't work for you, can we hold off of dropping support for the mixed install until there is a working way of building Sage with conda that gives a Jupyter kernel (or if its simple enough, include the fix in this PR?)

[vbraun]

This breaks

$ ./bootstrap -s
[...]
./bootstrap: installing /home/release/Sage/build/pkgs/sagemath_repl/src/requirements.txt
./bootstrap: installing /home/release/Sage/build/pkgs/sagemath_sirocco/src/pyproject.toml
./bootstrap: installing /home/release/Sage/build/pkgs/sagemath_sirocco/src/requirements.txt
./bootstrap: installing /home/release/Sage/build/pkgs/sagemath_tdlib/src/pyproject.toml
./bootstrap: installing /home/release/Sage/build/pkgs/sagemath_tdlib/src/requirements.txt
./bootstrap-conda:84: generate conda environment files
configure.ac:64: installing 'config/compile'
configure.ac:64: installing 'config/config.guess'
configure.ac:64: installing 'config/config.sub'
configure.ac:50: installing 'config/install-sh'
configure.ac:50: installing 'config/missing'
Creating upstream/configure-00a590f0e20f86b7fbcc2d8d1920458b848cbe80.tar.gz...
find: warning: you have specified the global option -maxdepth after the argument -name, but global options are not positional, i.e., -maxdepth affects tests specified before it as well as those specified after it.  Please specify global options before other arguments.
tar: src/environment-3.[89].yml: Cannot stat: No such file or directory
tar: src/environment-3.1[0-9].yml: Cannot stat: No such file or directory
tar: environment-optional-3.[89].yml: Cannot stat: No such file or directory
tar: environment-optional-3.1[0-9].yml: Cannot stat: No such file or directory
tar: src/environment-optional-3.[89].yml: Cannot stat: No such file or directory
tar: src/environment-optional-3.1[0-9].yml: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors

[tobiasdiez]

@tobiasdiez I'm not able to get a non-editable conda install working. It builds but some basic functionality is broken. For instance:

P2.<X, Y, Z> = ProjectiveSpace(2, QQ)
C = Curve(X^3 + Y^3 + Z^3)

is giving me

AttributeError: 'sage.rings.polynomial.multi_polynomial_libsingular.MPolynomialRing_libsingular' object has no attribute '_CategoryObject__gens_dict'

whereas I should be able to print(C) and get Projective Plane Curve over Rational Field defined by X^3 + Y^3 + Z^3.

Can you confirm whether the non-editable conda install works for you? If it does, can you add to the documentation the steps to do a non-editable install? If it doesn't work for you, can we hold off of dropping support for the mixed install until there is a working way of building Sage with conda that gives a Jupyter kernel (or if its simple enough, include the fix in this PR?)

Strange. I've never had troubles with the non-editable install. Is the error above the only one that you experience? Does it work for you if you use make but with non-editable install enforced (by default make uses the editable install if I remember correctly). (Maybe it's best to continue this discussion in a new issue.)

[tobiasdiez]

This breaks

$ ./bootstrap -s
[...]
./bootstrap: installing /home/release/Sage/build/pkgs/sagemath_repl/src/requirements.txt
./bootstrap: installing /home/release/Sage/build/pkgs/sagemath_sirocco/src/pyproject.toml
./bootstrap: installing /home/release/Sage/build/pkgs/sagemath_sirocco/src/requirements.txt
./bootstrap: installing /home/release/Sage/build/pkgs/sagemath_tdlib/src/pyproject.toml
./bootstrap: installing /home/release/Sage/build/pkgs/sagemath_tdlib/src/requirements.txt
./bootstrap-conda:84: generate conda environment files
configure.ac:64: installing 'config/compile'
configure.ac:64: installing 'config/config.guess'
configure.ac:64: installing 'config/config.sub'
configure.ac:50: installing 'config/install-sh'
configure.ac:50: installing 'config/missing'
Creating upstream/configure-00a590f0e20f86b7fbcc2d8d1920458b848cbe80.tar.gz...
find: warning: you have specified the global option -maxdepth after the argument -name, but global options are not positional, i.e., -maxdepth affects tests specified before it as well as those specified after it.  Please specify global options before other arguments.
tar: src/environment-3.[89].yml: Cannot stat: No such file or directory
tar: src/environment-3.1[0-9].yml: Cannot stat: No such file or directory
tar: environment-optional-3.[89].yml: Cannot stat: No such file or directory
tar: environment-optional-3.1[0-9].yml: Cannot stat: No such file or directory
tar: src/environment-optional-3.[89].yml: Cannot stat: No such file or directory
tar: src/environment-optional-3.1[0-9].yml: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors

Sorry, should be fixed now. Strange that it wasn't caught by the ci.

[vincentmacri]

Strange. I've never had troubles with the non-editable install.

Am I correct in assuming that to do a non-editable conda install you just need to change pip install --no-build-isolation --config-settings editable_mode=compat -v -v --editable ./src to pip install --no-build-isolation -v -v ./src, or is it different?

Is the error above the only one that you experience?

I've gotten this error message (or perhaps a similar one) doing a few curve-related things. Not sure if it's specific to curves, but I mainly use Sage for curve stuff and so most of what I'm doing is curve-related.

Does it work for you if you use make but with non-editable install enforced (by default make uses the editable install if I remember correctly).

What would be the steps to do that?

(Maybe it's best to continue this discussion in a new issue.)

I'll do some more testing once this is merged in case something you did to conda here fixes it (I was trying to do the non-editable install on develop I think). If I'm still having trouble I'll open a new issue. I'd also want to figure out how to do the non-editable make install you mentioned and test that so I can include the results of that in my bug report.

[dimpase]

check the options of ./configure for this. Then

make sagelib-clean sagelib-uninstall 
./configure [with the correct options]
make build 

[tobiasdiez]

Strange. I've never had troubles with the non-editable install.

Am I correct in assuming that to do a non-editable conda install you just need to change pip install --no-build-isolation --config-settings editable_mode=compat -v -v --editable ./src to pip install --no-build-isolation -v -v ./src, or is it different?

Exactly! You might also try the new meson-based built system (which will be the default hopefully very soon) by using pip install --no-build-isolation -v -v . (i.e. use the pyproject.toml file located in the root)

[user202729]

As a side note, you didn't fix https://doc-pr-38728--sagemath.netlify.app/html/en/installation/meson to say environment-dev-... instead of src/environment-dev-... . (first code block)