avoiding overlay of production and development denvs

[tomeichlersmith]

I've seen the following confusion happen frequently (at least twice to my knowledge over the last few weeks which is too high in my opinion). The person ends up in a state where their filesystem looks like

- path/to/ldmx/
  - .denv/      # development denv created by running just init
  - ldmx-sw/
    - .denv/    # production denv created by running denv init ldmx/pro:<version> within ldmx-sw

This leads to the very confusing situation where denv fire can work from within ldmx-sw even if its not built (since it reads from the production image denv) while just fire will not work since it is using the development image and ldmx-sw is not built yet.

Short-Term Solution

If you find yourself in this state, delete the production image denv within ldmx-sw and (if you still need it), re-create it in another directory.

rm -r ldmx-sw/.denv
mkdir my-other-project
cd my-other-project
denv init ldmx/pro:<version>

Long-Term Solution

The solution is to keep these two denvs separate. Literally any other directory other than ldmx-sw can be used for the ldmx/pro denv so maybe this can be resolved by updating the notes on the website to emphasize this point?

We could update the init recipe to specifically make sure that no denv is already created erroneously within ldmx-sw.

https://github.com/LDMX-Software/ldmx-sw/blob/6c08a49cb7a0bd8cff03f3e838f91f0536261c99/justfile#L85-L100

I'd like to avoid a hardcoded check within denv since I use it for other (non-LDMX or ldmx-sw) projects. Another idea is to avoid keeping the development denv outside of ldmx-sw. This choice is a relic of the ldmx suite of bash functions and could be avoided with how denv is designed. Putting the ldmx-sw denv within ldmx-sw would then help avoid this issue - you would need to overwrite the denv in ldmx-sw to get into a broken state like this and denv warns you before you would do so.

[tomeichlersmith]

For the medium term, I've added a warning to the getting started area informing anyone getting started to avoid running denv init within ldmx-sw itself.

https://github.com/LDMX-Software/ldmx-software.github.io/commit/ca56d1f90f11a64184ca492ffdc39fc1867d25b3

[tvami]

I'm cross posting from https://github.com/LDMX-Software/ldmx-sw/issues/1505#issuecomment-2658269267

This happened to one of my new student now

In the long term, we should decide if this is a workflow that should be supported. Having denvs within one another can easily lead to confusion since cd .. can change which denv is chosen to run. If we do want to support this workflow, then we would want to pass the prompts that denv issues to the user so they are at least acknowledging that they have nested denvs.

What I'd like is to have a warning that there is a .denv above, and the option to get rid of it.

This is what he saw

cd /home/henry/ldmx/noise_study/ldmx-sw/
just init 
/home/henry/.local/bin/denv: line 495: /home/henry/ldmx/noise_study/.denv/config: No such file or directory
error: Recipe `init` failed with exit code 1

while he had a .denv in

cd /home/henry/ldmx/.denv/

Deleting /home/henry/ldmx/.denv/ made it work, but it would be good to streamline this for newcomers

[tomeichlersmith]

I kind of snuck this into my first post, but I want to bring it up again:

Putting the ldmx-sw denv within ldmx-sw would then help avoid this issue

I think transitioning to this working mode is pretty simple, it requires changing one line in the environment initialization of the container.

https://github.com/LDMX-Software/dev-build-context/blob/ffc71612016287ac04ffbe3adbf5cb9236bb2f3c/ldmx-env-init.sh#L17

and updating the init recipe

https://github.com/LDMX-Software/ldmx-sw/blob/615bde3eb3e57c7b7f2343b02adbfcd000448a11/justfile#L128-L133

In order to make this move as smooth as possible, I think we need to do a few things. Just brainstorming here...

• Make sure no one is using the ldmx bash suite of functions anymore. #1716
• Scripting in ldmx-env-init.sh to support both locations
  • Probably just checking on if ldmx-sw is a subdirectory of ${HOME} or not
  • https://github.com/LDMX-Software/dev-build-context/pull/138 and the following release https://github.com/LDMX-Software/dev-build-context/releases/tag/v5.1.1
• Update just init to have denv_workspace be the ldmx-sw repository root
• Drop support for both locations in ldmx-env-init.sh

Biggest question mark for me is if people use files "parallel" to ldmx-sw:

ldmx/       # ldmx-sw parent directory, aka LDMX_BASE, used as denv_workspace\
|- ldmx-sw/  # actual clone of ldmx-sw that is being developed
|- config.py # some file "outside" ldmx-sw but inside ldmx you want to use with your developed ldmx-sw
|- my-project/config.py # another example of "parallel"
other-dir/file.root # not parallel because person would need to denv config mounts even now

This is the only situation where special care would need to be taken in order to still be able to run config.py with the locally-built ldmx-sw. Files inside ldmx-sw (like the source code itself) would not need to be moved at all and files outside LDMX_BASE (ldmx in example above) already need special care so folks would still need to denv config mounts like they do now.

[tvami]

Biggest question mark for me is if people use files "parallel" to ldmx-sw:

My 2 cents is that we should not support this. Instead just document it clearly that whenever there is an ldmx-sw, the work should be done under that.

[tomeichlersmith]

I agree, that's what we do now (just with the directory everything is "under" is the ldmx-sw parent directory aka LDMX_BASE and not ldmx-sw itself): tell folks to put work under a specific directory and if they can't (e.g. with large data files residing somewhere else), they need to denv config mounts.

I'm more worried about how prevalent the parallel files are currently. If a lot of folks usually have files parallel to ldmx-sw right now, the transition will be more work than if only a few do. My concern isn't really about what should be supported but how difficult the transition will be. Maybe I can mention it at the next SW dev meeting to take a loose poll.

[tomeichlersmith]

I can play around with moving the denv into ldmx-sw without having to modify the ldmx/dev image.

Instead of setting LDMX_BASE to the HOME directory, I am setting the LDMX_SW_INSTALL environment variable directly.

This isn't the long term solution I would like, but it should work.

Forcibly create override denv within ldmx-sw

cd ldmx-sw
denv init --clean-env --name ldmx ldmx/dev:latest
# answer Y to override question

Change to .profile

28,31c28
< # make sure LDMX_BASE is defined for ldmx-env-init.sh
< if [ -z "${LDMX_BASE+x}" ]; then
<   export LDMX_BASE="${HOME}"
< fi
---
> export LDMX_SW_INSTALL="${HOME}/install"

[tomeichlersmith]

The 1472 branch attached starts this editing process. Found out that moving the denv into ldmx-sw means that we can use git workspace which I find very helpful.

Still To Do

• [x] check what using the 1472 branch looks like with the parent directory being the denv_workspace
• [x] what happens when re-just init
• [x] separate out necessary changes from breaking changes if the two denv styles are mutually exclusive. As far as I can tell, there are no breaking changes.
• [x] do we need to update the image? Yes, to avoid annoying env var tracking.

[tomeichlersmith]

• just init should error out, folks should remove the parent denv if they want to re-init
• just configure should work well until we want to abandon support for parent denv

Script started on 2025-07-28 13:04:42-05:00 [TERM="xterm-256color" TTY="/dev/pts/0" COLUMNS="118" LINES="66"]
tom@appa:~/code/ldmx/ldmx-sw$ git ⌫⌫⌫⌫♪♪♪♪♪♪denv config print
↵
denv_workspace="/home/tom/code/ldmx"
denv_name="ldmx"
denv_image="ldmx/dev:v5.0.1"
denv_mounts=""
denv_shell="/bin/bash -i"
denv_network="true"
podman version 4.9.3
LDMX_RUN_NUMBER=1
LDMX_NUM_EVENTS=10
XAUTHORITY=/home/tom/.Xauthority
DISPLAY=:0
DENV_NAME=ldmx
DENV_RUNNER=podman
DENV_IMAGE=ldmx/dev:v5.0.1
HOME=/home/tom/code/ldmx
tom@appa:~/code/ldmx/ldmx-sw$ just init
↵
ERROR: denv cannot init into '/home/tom/code/ldmx/ldmx-sw' without overriding '/home/tom/code/ldmx'.
error: Recipe `init` failed with exit code 1
tom@appa:~/code/ldmx/ldmx-sw$ just configure
↵
git fetch --tags && git describe --tags | cut -f 1 -d '-' > VERSION
git rev-parse HEAD > COMMIT_SHA
denv cmake -B build -S . -DADDITIONAL_WARNINGS=ON -DENABLE_CLANG_TIDY=ON 
[ INFO ]: Building all modules.
[ INFO ]: Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.83.0/BoostConfig.cmake (found version "1.83.0") found components: log 
[ INFO ]: Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.83.0/BoostConfig.cmake (found version "1.83.0") found components: system log filesystem thread chrono atomic regex 
[ INFO ]: Deduced git SHA: 1b018f895c534f1fa9013ec855162235a849e798
[ INFO ]: Building ROOT dictionary LinkDef
Looking for libboost_python
[ INFO ]: Building the modules necessary for the reconstruction.
[ INFO ]: Found ONNX Runtime version 
[ INFO ]: Found Acts 36.0.0
[ INFO ]: Found Geant4 version 10.2.3
[ INFO ]: Building the modules necessary for the simulation.
[ INFO ]: Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.83.0/BoostConfig.cmake (found version "1.83.0") found components: log 
[ INFO ]: Found GENIE
[ INFO ]: Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.83.0/BoostConfig.cmake (found suitable version "1.83.0", minimum required is "1.68") found components: iostreams 
[ INFO ]: Installing all field map files.
-- Configuring done (0.7s)
-- Generating done (0.6s)
-- Build files have been written to: /home/tom/code/ldmx/ldmx-sw/build
tom@appa:~/code/ldmx/ldmx-sw$ denv printenv LDMX_SW_INSTALL
↵
/home/tom/code/ldmx/ldmx-sw/install
tom@appa:~/code/ldmx/ldmx-sw$ exit
↵
exit

Script done on 2025-07-28 13:05:14-05:00 [COMMAND_EXIT_CODE="0"]