Cannot use env vars to point to other Conda env with running workers

[visr]

In trying to debug #276 what confused me for a while was that I got errors like this:

ERROR: Julia server returned error after receiving "run" command:
Failed to run notebook: C:\ProgramData\DevDrives\repo\ribasim\Ribasim\docs\guide\delwaq.qmd
ERROR: EvaluationError: Encountered 13 errors during evaluation

Error 1 of 13
@ C:\ProgramData\DevDrives\repo\ribasim\Ribasim\docs\guide\delwaq.qmd:37
Python: ModuleNotFoundError: No module named 'ribasim'
Python stacktrace:
 [1] <module>
   @ C:\ProgramData\DevDrives\repo\ribasim\Ribasim\docs\guide\delwaq.qmd:38

While I was running quarto from environment B I noticed that QuartoNoteBookRunner still had A as the active environment. I ran from A earlier. But I didn't realize that QuartoNotebookRunner kicks of a server that stays alive, and needs to be manually killed such that it will start a new server for environment B.

I'm not sure what can be done about this, but ideally the server gets restarted in such a case if possible.

[MichaelHatherly]

There is some work that @jkrumbiegel is doing in that general direction in https://github.com/quarto-dev/quarto-cli/pull/11803 that may help this situation a bit.

[MichaelHatherly]

While I was running quarto from environment B I noticed that QuartoNoteBookRunner still had A as the active environment.

How had you set the environment? Via --project in the julia.exeflags frontmatter, or some other way?

[visr]

I ran quarto from a pixi environment. This starts up the right server, but if you then try to run quarto from another pixi environment the server will not be configured correctly.

https://github.com/visr/qnr-plot-mwe/blob/2ab6a65ec72d89f757177d6bdc49d070da48b55f/pixi.toml

[jkrumbiegel]

There's a difference between the environment that the main server process runs with and the environment that a given worker process runs with. I'm not sure I understood what you meant by

While I was running quarto from environment B I noticed that QuartoNoteBookRunner still had A as the active environment.

And in general the main server should almost never need to be killed, it's generally the workers that you'd want to restart if something goes wrong with them. But we don't have an interface via quarto for that, yet. The PR Michael linked is a start.

[visr]

Ah I see, so probably the worker environments are the issue. And specifically the environment variables that Conda uses to point to the right environment.

Specifically what I observed was that all of these were still pointing to Conda environment A when running QuartoNotebookRunner from environment B, and only after killing julia processes this was resolved since it spun up new workers with different environment variables.

[jkrumbiegel]

Ah yes env variables are a special case because I think they're inherited from the server process, but that spins up only once. But there's also a way to set env vars via frontmatter if I don't misremember

[visr]

That would make sense. There are some examples of env vars via frontmatter in https://github.com/PumasAI/QuartoNotebookRunner.jl/issues/217.

[mcanouil]

As it seems related, I'm posting a small example where the Julia engine does not have access to environment variables while the other engine do.

WHATEVER="something else" quarto render index.qmd

---
engine: julia
---

```{julia}
println(ENV["WHATEVER"])
```

It also does not have access to _environement (https://quarto.org/docs/projects/environment.html).

Original discussion:

• https://github.com/quarto-dev/quarto-cli/discussions/13289