Struggling with permissions issues for the AllUsers settings on Windows

[moorepants]

Checklist

• [X] I added a descriptive title
• [X] I searched open reports and couldn't find a duplicate

What happened?

We build a constructor Windows installer annually and install for AllUsers in silent mode. This year many/most of the installed scripts in <install-directory>/Scripts/ have permissions that prevent execution. For example, opening the Anaconda prompt and typing Jupyter lab gives Error executing Jupyter command 'lab': [WinError 5] Access is denied. Similar errors are produced when trying to launch lab from the Anaconda Navigator. Running spyder from the prompt or Navigator works. We tried building the new installer with the constructor version (3.4.5) we used last year but still get the permission errors. We have many details of our trial and error in this issue: https://gitlab.ewi.tudelft.nl/bhmgerritsen/anaconda-dee-config/-/issues/64 but I'll summarize below.

Constructor setup:

• Channel: conda-forge (with anaconda-navigator, console_shortcut, menuinst, powershell_shortcut using defaults:: notation and with conda channel priority set to "strict". The idea was to install everything form conda-forge except for these few anaconda specific packages.
• We install with silent mode like tudelft-anaconda-v2024.01-Windows-x86_64.exe /InstallationType=AllUsers /RegisterPython=1 /S /D=C:\Programs\tudelft-anaconda (but also have tried graphical installs)
• construct.yaml is here: https://gitlab.ewi.tudelft.nl/bhmgerritsen/anaconda-dee-config/-/blob/master/construct.yaml
• Our latest installer is here: https://gitlab.ewi.tudelft.nl/bhmgerritsen/anaconda-dee-config/-/releases

If we download the latest official Anaconda windows installer and run it in silent mode the permissions are correct. But our installer leaves many files with restricted permissions.

Changes from last year:

• The approach was have channel priority with defaults then conda-forge and install the anaconda package and then add various extra packages on top of anaconda from conda-forge.
• We set the channel_priority=strict this year and didn't last year.
• We hard pin the versions of all packages this year.
• Our constructor files from last year's release are here: https://gitlab.ewi.tudelft.nl/bhmgerritsen/anaconda-dee-config/-/tree/62d723fbbfabe495a0bb38c408b51a40dd1a8adc

We have run out of ideas as to what causes the permissions errors. Do you have any suggestions for what we may be doing wrong?

Conda Info

NA

Conda Config

NA

Conda list

NA

Additional Context

No response

[marcoesters]

I downloaded your 2024.01 and ran the GUI installation - everything looks good on my end. jupyter lab launches from the prompt without problems. I ran it on a Windows 11 machine where I'm the only user.

The fact that this also breaks with the old constructor makes me think that there is a specific interaction with your system. Does it maybe restrict what unsigned installers and/or binaries can do? The Anaconda and, since recently, the binaries on Windows are all signed by Anaconda.

You could also check if installing the package from defaults instead of conda-forge helps.

[moorepants]

Thanks @marcoesters, good to know that it worked for you. We can look more closely at things in the environment we are installing to. I'll ask out IT if we need the signing and we're already working on checking the installs from defaults. I'll report back on our success/failure.

[moorepants]

@marcoesters did you do a "JustMe" installation? Because that does seem to work for us. It is the "AllUsers" option that has the borked permissions.

[moorepants]

One other important note I forgot from above, is that the installer we built last year works just fine with and AllUsers install on the computers this year. So that is odd.

[subsense33]

To reproduce the error, it is important to start the installer as the local SYSTEM account, not just as a local administrator. Our software deployment mechanism also installs software using the local SYSTEM user.

You can use Microsoft psexec.exe to start an elevated command prompt as the SYSTEM user.

1. Open an elevated Command Prompt.
2. Run the following command: psexec.exe -s -i cmd
3. A new Command Prompt window will open. To verify your current user, type whoami.

[moorepants]

To reproduce the error, it is important to start the installer as the local SYSTEM account

Does that mean you have reproduced the error? This is exactly how we are installing using the SYSTEM account.

Oddly, the installer we build last year installs with correct permissions with the SYSTEM account but the new installer (built with either constructor 3.4.5 or the latest version) has incorrect permissions with the SYSTEM account.

[moorepants]

Ah, I didn't realize @subsense33 was my colleague here at TU Delft. He's just explaining what we've done. I assumed @subsense33 was another person commenting on this.

[moorepants]

I just installed the v2024.01 release from https://gitlab.ewi.tudelft.nl/bhmgerritsen/anaconda-dee-config/-/releases on a regular Windows PC. I selected the "All Users" option and to "Register Python". I elevated admin rights to run the installer and then afterwards checked running spyder, jupyter, python console, ipython both from command line and anaconda navigator app and everything worked correctly with no permissions issues. This is presumably what @marcoesters did.

So, this points towards some kind of issue installing with the local SYSTEM account. (except that last year's installer has no issue installing with the SYSTEM account)

[marcoesters]

Thanks @subsense33 for these instructions! I was able to reproduce the problem now, but I suspect this is not related to constructor or your installer, but to jupyterlab. Here is what I did to come to that conclusion:

• Open Anaconda Prompt
• Run jupyter lab --> WinError 5. I also ran jupyter notebook with the same issue. Other executables like spyder open without errors.
• Create an environment in a location writable to me: conda create -p %USERPROFILE%\testenv
• conda activate %USERPROFILE%\testenv
• conda install -y jupyterlab -c conda-forge
• Run jupyter lab --> jupterlab opens without error

My hypothesis is that jupyterlab tries to write files into its own install location, which is write-protected. This would also explain why rebuilding the installer with an older constructor version doesn't work either. Your installer from last year (I assume) contains an older version of jupyterlab, which may not exhibit this behavior.

[marcoesters]

Of course, executing jupyterlab from the new environment means that I have full permissions to that executable as well. However, the fact that I can execute all other binaries in the base environment tells me that jupyter tries to do some kind of write operation into a location that it's not allowed to do.

[marcoesters]

I also manually checked the permissions inside the Scripts folder and they are what I expect: as a user, I only have read and execute permissions. The WinError is also a python error, so it reaches and executes the python interpreter somehow. If this was a permission error of your directory, I'd expect the error to be emitted by Windows, not python.

[moorepants]

Thanks for the insight. The permission error also occurs when trying to run ipython from the anaconda prompt.

We could install an older version of IPython and Jupyter as a stopgap.

[moorepants]

Here is an issue on the IPython tracker that implies that IPython tries to create something in the user's home directory:

https://github.com/ipython/ipython/issues/12079

It is from 2020 though, so that doesn't align with our last year's installer working.

[moorepants]

Searching for "jupyter" and "access is denied" gives lots of hits, like: https://stackoverflow.com/questions/61747187/jupyter-lab-in-anaconda-not-working-error-executing-jupyter-command-lab-win

[moorepants]

It's worth noting that we see this:

indicating that we don't even have read and execute permissions for all the binaries.

[marcoesters]

You are right, alembic and other binaries don't have the correct permissions. jupyter does though.

For AllUsers installations, the permissions are supposed to be fixed here: https://github.com/conda/constructor/blob/main/constructor/nsis/main.nsi.tmpl#L1240-L1253

You can monitor the installation and see that you will lose read access to the entire directory during the installation, but get it back shortly after. Maybe the permissions changes are not performed recursively? I will have to investigate what those commands do exactly, but that change is also two years old, so this cannot be the only factor.

Alembic doesn't have read access throughout the entire installation process, even before the permissions are fixed in the lines above.

You may be running into a few things at once here:

• ~jupyter has the correct permissions, so you may see the problem I described earlier. This isn't something constructor will be able to change, but maybe you can configure jupyter/ipython in a post-install script to write somewhere else.~ (the second point is more likely)
• jupyter calls another binary that doesn't have the correct permissions. I don't know enough about the inner workings of jupyter though.
• As for alembic et al., constructor may have to retroactively change the permissions recursively if this isn't done already. I will have to think about whether this comes with security risks, even though read and execute privileges should be relatively safe.

[moorepants]

I will have to think about whether this comes with security risks, even though read and execute privileges should be relatively safe.

Our current work around is to give everything in <install-dir>/Scripts read/exec permissions in a post install script.

[subsense33]

I will have to think about whether this comes with security risks, even though read and execute privileges should be relatively safe.

Our current work around is to give everything in <install-dir>/Scripts read/exec permissions in a post install script.

Yesterday I changed the work-around script to reset the owner on <install-dir>. Not just <install-dir>/Scripts. Does no harm, just to be sure everything is set correctly. The script resets the owner of the entire tree to the local group "Administrators", that is how it's supposed to be. (Like the Anaconda deployment of past years).

Normal users have only read and execute permissions, that is safe. So something weird is happening with the "owner" of certain files. I use the Alembic binary just to check if the problem is there, but more files are affected.

[marcoesters]

While that workaround probably works, the ownership of the files is not different from the Anaconda installer. I am using jupyter-kernel.exe as an example here. File permissions are listed using Get-Acl | fl.

Anaconda installer, installed as SYSTEM:

Path   : Microsoft.PowerShell.Core\FileSystem::C:\ProgramData\anaconda3\Scripts\jupyter-kernel.exe
Owner  : BUILTIN\Administrators
Group  : NT AUTHORITY\SYSTEM
Access : BUILTIN\Users Allow  ReadAndExecute, Synchronize
         NT AUTHORITY\SYSTEM Allow  FullControl
         BUILTIN\Administrators Allow  FullControl
Audit  :
Sddl   : O:BAG:SYD:AI(A;ID;0x1200a9;;;BU)(A;ID;FA;;;SY)(A;ID;FA;;;BA)

TU Delft installer, installed as SYSTEM:

Path   : Microsoft.PowerShell.Core\FileSystem::C:\ProgramData\tudelft-anaconda\Scripts\jupyter-kernel.exe
Owner  : BUILTIN\Administrators
Group  : NT AUTHORITY\SYSTEM
Access : NT AUTHORITY\SYSTEM Allow  FullControl
         BUILTIN\Administrators Allow  ReadAndExecute, Synchronize
Audit  :
Sddl   : O:BAG:SYD:PAI(A;;FA;;;SY)(A;;0x1200a9;;;BA)

So, the owner is the same, but for some reason, access isn't granted the same way. I'm wondering if this is because of differences between the two build systems (Anaconda vs. conda-forge).

I'd be really curious to see if the problem persists if you build the installer using only defaults packages.

[moorepants]

This was my attempt at building the new installer the same way we did last year (defaults as priority w/ conda forge packages on top):

https://github.com/moorepants/anaconda-dee-config/actions/runs/10185441100

We have the same permissions issue with it.

I can build an installer without any packages from conda forge, but I just have to go through an make a new construct.yaml that eliminates all packages from conda forge. Last year's approach included the anaconda package (as does the one I linked just above). Are you suggesting to build an installer only from defaults package but without using the anaconda package or to include it?

Note that I switched to the "conda forge only" approach because it has long been recommend to not mix conda forge packages with defaults packages by conda forge. Our users need packages only available on conda forge, so I went the "pure" conda forge route to ensure binary compatibility.

[marcoesters]

I suggest to not use conda-forge as a channel for this test, yes. I'd also reduce the number of packages since we know which packages are problematic: the packages that install alembic or jupyter-kernel. That just makes debugging faster.

If that results in the same problem, I'm out of ideas. The workflow looks okay. The only difference I see is that you use PowerShell instead of bash (as Miniforge and Anaconda do), but that can't be the reason, can it?

[moorepants]

I built an installer without conda-forge packages in the base environment. You can download here:

https://github.com/moorepants/anaconda-dee-config/actions/runs/10242877472

Note that my install action fails with:

Error: Unable to locate executable file: C:\Users\runneradmin\miniconda3\condabin\conda.bat. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also verify the file has a valid extension for an executable file.

I'm not sure why. I did not include the anaconda package.

[moorepants]

PowerShell instead of bash (as Miniforge and Anaconda do), but that can't be the reason, can it?

I can test a build using bash to call the constructor related commands.

[marcoesters]

I built an installer without conda-forge packages in the base environment. You can download here:

https://github.com/moorepants/anaconda-dee-config/actions/runs/10242877472

Note that my install action fails with:

Error: Unable to locate executable file: C:\Users\runneradmin\miniconda3\condabin\conda.bat. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also verify the file has a valid extension for an executable file.

I'm not sure why. I did not include the anaconda package.

Probably because the installation has failed. I'd run the GUI with the artifact and/or create an environment with your specs to see if there is a package conflict.

[Dribbelswag]

@marcoesters and @moorepants Jason made an (test) installer for the upcoming academic year. He used Constructor 3.11.3

But the issue persists.

To reproduce the error, it is important to start the installer as the local SYSTEM account, not just as a local administrator. Our software deployment mechanism also installs software using the local SYSTEM user. (SCCM/SCM and Intune)

You can use Microsoft psexec.exe to start an elevated command prompt as the SYSTEM user.

1. Open an elevated Command Prompt.
2. Run the following command: psexec.exe -s -i cmd
3. A new Command Prompt window will open. To verify your current user, type whoami.

Please let me know if there is anything I can do to help solve this issue. I do not have in-depth knowledge of Conda / Constructor, @moorepants does. I do have in-depth knowledge of de Windows OS's, Application Packaging and deployment.

I assume this is the step where the permissions get messed up:

Note; subsense33 and Dribbelswag are the same person. me ;-) Just two different accounts. Sorry for the confusion.

[moorepants]

This is the new installer we build this year with Constructor 3.11.3:: https://github.com/moorepants/anaconda-dee-config/actions/runs/14701742124/artifacts/3019641164

[moorepants]

Looking at the conversation last year, I guess we still need to make an installer with a minimal set of packages and see if it also has the permissions issues.

One thing I just thought of. It might be that the last working installer 2023.01 (without the permissions issues) was built on one of our university Windows desktops and the ones with the permissions issues are the ones I build in the Github Actions CI. I will look into this.

[moorepants]

There was also this issue https://github.com/conda/constructor/issues/872 that is more recent that is also related to permissions. May be worth comparing this issue and that one.

[moorepants]

I see that @Dribbelswag did build the 2023 and 2024 versions on an identical computer (not GIthub CI) here: https://gitlab.ewi.tudelft.nl/bhmgerritsen/anaconda-dee-config/-/issues/64#note_2408814 The errors occur in the 2024+ versions whether we build locally or on CI.

[marcoesters]

There was also this issue https://github.com/conda/constructor/issues/872 that is more recent that is also related to permissions. May be worth comparing this issue and that one.

I am not sure those are related since your is specific to the SYSTEM account.

Looking at the conversation last year, I guess we still need to make an installer with a minimal set of packages and see if it also has the permissions issues.

Have you tried an installer that only uses packages from defaults instead of conda-forge? That's the big difference I can see that may affect permissions. That doesn't mean that the issue isn't constructor, but it would give us a hint. Anaconda Navigator seems to work as opposed to the other binaries.

Note that my availability for the next two weeks will be very spotty, so apologies in advance if replies are delayed.