PythonCall.jl icon indicating copy to clipboard operation
PythonCall.jl copied to clipboard
verified publisher icon JuliaPy

Completely disable outside python

Open Moelf opened this issue 3 years ago • 26 comments 

 Package         Version  Build               Channel           Size
───────────────────────────────────────────────────────────────────────
  Reinstall:
───────────────────────────────────────────────────────────────────────

  o libstdcxx-ng   11.3.0  h239ccf8_18         conda-forge     Cached
  o python         3.10.6  h582c2e5_0_cpython  conda-forge     Cached

  Summary:

  Reinstall: 1 packages

  Total download: 0 B

───────────────────────────────────────────────────────────────────────


Transaction starting
Changing libstdcxx-ng-11.3.0-h239ccf8_18 ==> libstdcxx-ng-11.3.0-h239ccf8_18
Changing python-3.10.6-h582c2e5_0_cpython ==> python-3.10.6-h582c2e5_0_cpython
Transaction finished
Python path configuration:
  PYTHONHOME = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt'
  PYTHONPATH = '/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/itk:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/python:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages'
  program name = '/home/jiling/.julia/dev/WVZAnalysis/.CondaPkg/env/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/home/jiling/.julia/dev/WVZAnalysis/.CondaPkg/env/bin/python'
  sys.base_prefix = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt'
  sys.base_exec_prefix = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt'
  sys.platlibdir = 'lib'
  sys.executable = '/home/jiling/.julia/dev/WVZAnalysis/.CondaPkg/env/bin/python'
  sys.prefix = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt'
  sys.exec_prefix = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt'
  sys.path = [
    '/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/itk',
    '/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/python',
    '/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib',
    '/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages',
    '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt/lib/python310.zip',
    '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt/lib/python3.10',
    '/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding

The outside environment setup a pretty intrusive set of environment variables, how to protect against it?

Moelf avatar Oct 19 '22 20:10 Moelf

current CondaPkg.toml is

  [deps]
  python = ">=3.10,<4"
  uproot = ""
  hist = ""

Moelf avatar Oct 19 '22 21:10 Moelf 

ERROR: LoadError: InitError: failed process: Process(setenv(`/home/jiling/.julia/dev/WVZAnalysis/.CondaPkg/env/bin/python 
/home/jiling/.julia/packages/PythonCall/eU0yr/src/cpython/find_libpython.py --list-all`,["OpenLoopsPath=/cvmfs/sft.cern.ch
/lcg/releases/MCGenerators/openloops/2.1.2-8fff2/x86_64-centos7-gcc11-opt", "PATH=/cvmfs/sft-nightlies.cern.ch/lcg/views
/dev4/Wed/x86_64-centos7-gcc11-opt/scripts:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/bin:
/cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0-ad0f5/x86_64-centos7/bin:/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-
centos7/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/opt/dell/srvadmin/bin:/home/jiling/.fzf/bin:
/home/jiling/.local/bin:/home/jiling/bin", "EDITOR=vim", "LHAPDF_DATA_PATH=/cvmfs/sft.cern.ch/lcg/external/lhapdfsets/current
/:/cvmfs/sft-nightlies.cern.ch/lcg/latest/MCGenerators/lhapdf/6.5.3-d2254/x86_64-centos7-gcc11-opt/share/LHAPDF:", 
"CXX=/cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0-ad0f5/x86_64-centos7/bin/g++", "CC=/cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0-
ad0f5/x86_64-centos7/bin/gcc", "LD_LIBRARY_PATH=/cvmfs/sft-nightlies.cern.ch/lcg/latest/MCGenerators/thepeg/2.2.3-9e086
/x86_64-centos7-gcc11-opt/lib/ThePEG:/cvmfs/sft-nightlies.cern.ch/lcg/latest/MCGenerators/herwig++/7.2.3-54ceb/x86_64-
centos7-gcc11-opt/lib/Herwig:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-
packages/jaxlib/mlir/_mlir_libs:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-
packages/torch/lib:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
/onnxruntime/capi/:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
/tensorflow:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/tensorflow
/contrib/tensor_forest:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
/tensorflow/python/framework:/cvmfs/sft.cern.ch/lcg/releases/java/8u312-80070/x86_64-centos7-gcc11-opt/jre/lib/amd64:/cvmfs
/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib64:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed
/x86_64-centos7-gcc11-opt/lib:/cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0-ad0f5/x86_64-centos7/lib:/cvmfs/sft.cern.ch/lcg/releases
/gcc/11.3.0-ad0f5/x86_64-centos7/lib64:/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/lib:/cvmfs/sft-
nightlies.cern.ch/lcg/latest/R/4.1.2-63468/x86_64-centos7-gcc11-opt/lib64/R/library/readr/rcon", "GOBIN=/cvmfs/sft-
nightlies.cern.ch/lcg/latest/gophernotes/7c3ca4e-d1acd/x86_64-centos7-gcc11-opt/bin", "DD4hepINSTALL=/cvmfs/sft-
nightlies.cern.ch/lcg/nightlies/dev4/Wed/DD4hep/01.23/x86_64-centos7-gcc11-opt", "MANPATH=/cvmfs/sft-nightlies.cern.ch
/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/man:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-
opt/share/man:/cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0-ad0f5/x86_64-centos7/share/man:/cvmfs/sft.cern.ch/lcg/releases/binutils
/2.37-355ed/x86_64-centos7/share/man::/opt/puppetlabs/puppet/share/man"  …  "GVBINDIR=/cvmfs/sft-nightlies.cern.ch
/lcg/latest/graphviz/2.40.1-0ed86/x86_64-centos7-gcc11-opt/lib/graphviz", "JULIA_LOAD_PATH=/home/jiling/.julia
/dev/WVZAnalysis/Project.toml:/home/jiling/.julia/environments/v1.7/Project.toml:/cvmfs/sft.cern.ch/lcg/releases/julia/1.7.3-
b9498/x86_64-centos7-gcc11-opt/share/julia/stdlib/v1.7:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-
opt/share/julia/environments/v1.7/Project.toml", "PKG_CONFIG_PATH=/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-
centos7-gcc11-opt/lib64/pkgconfig:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/pkgconfig", 
"ROOT_INCLUDE_PATH=/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/include/Geant4:/cvmfs
/sft.cern.ch/lcg/releases/jsonmcpp/3.10.5-f26c3/x86_64-centos7-gcc11-opt/include:/cvmfs/sft-nightlies.cern.ch/lcg/views
/dev4/Wed/x86_64-centos7-gcc11-opt/src/cpp:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/include:
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.12-9a1bc/x86_64-centos7-gcc11-opt/include/python3.9:/cvmfs/sft-nightlies.cern.ch
/lcg/latest/R/4.1.2-63468/x86_64-centos7-gcc11-opt/lib64/R/include:/cvmfs/sft-nightlies.cern.ch/lcg/latest/R/4.1.2-63468/x86_64-
centos7-gcc11-opt/lib64/R/library/RInside/include:/cvmfs/sft-nightlies.cern.ch/lcg/latest/R/4.1.2-63468/x86_64-centos7-gcc11-
opt/lib64/R/library/Rcpp/include", "DATA=/data/jiling", "G4ABLADATA=/cvmfs/geant4.cern.ch/share/data/G4ABLA3.1", 
"G4INSTALL=/cvmfs/sft-nightlies.cern.ch/lcg/latest/Geant4/11.0.3-78116/x86_64-centos7-gcc11-opt", 
"OPENBLAS_MAIN_FREE=1", "OCTAVE_HOME=/cvmfs/sft-nightlies.cern.ch/lcg/latest/octave/6.2.0-055df/x86_64-centos7-
gcc11-opt", "GOROOT=/cvmfs/sft.cern.ch/lcg/releases/go/1.12.5-ff8e6/x86_64-centos7-gcc11-opt"]), ProcessExited(1)) [1]

Moelf avatar Oct 19 '22 21:10 Moelf

this works

$ env PYTHONHOME= PYTHONPATH= julia --project=.

Moelf avatar Oct 19 '22 21:10 Moelf

Ah ok thanks, I need to make CondaPkg clear out any Python env vars.

cjdoris avatar Oct 20 '22 18:10 cjdoris

can that be done? if so yeah that should work; I was wondering if it's a CondaPkg bug

Moelf avatar Oct 20 '22 19:10 Moelf

Yep, you can do delete!(ENV, "PYTHONHOME") for example (it only affects the running process). CondaPkg already modifies other env vars.

cjdoris avatar Oct 20 '22 19:10 cjdoris

that doesn't work, the sub-processes spawned later gets envs from the original Julia process

Moelf avatar Oct 20 '22 19:10 Moelf

It does, if you do delete!(ENV, "PYTHONHOME") from Julia then start Python from within Julia, it won't see PYTHONHOME.

cjdoris avatar Oct 21 '22 07:10 cjdoris

sorry, I didn't mention that when I say it doesn't work, I already tried.

Moelf avatar Oct 21 '22 14:10 Moelf 

ENV["PYTHONHOME"] = ""
ENV["PYTHONPATH"] = ""
using PythonCall, Serialization

function make_TH1D(h)
    np = pyimport("numpy")
...
end

this fails in the middle of the CondaPkg trying to install the environment and I still see those environment variable

Moelf avatar Oct 21 '22 14:10 Moelf

bump

Moelf avatar Jan 26 '23 19:01 Moelf

I also don't quite understand why this wouldn't work

module A

if contains(ENV["PYTHONHOME"], "cern.ch/lcg/release")
    @info "Using LCG Python, if it doesn't work try delete `.CondaPkg/` folder"
     # use LCG release python
     ENV["JULIA_CONDAPKG_BACKEND"] = "Null"
end
using PythonCall
end

I tested this in a standalone PythonCall.jl it works, but not when used in another package?

Moelf avatar Jan 26 '23 19:01 Moelf

I can reproduce your issue, and have pushed a fix to main. Please try it out with pkg> add PythonCall#main.

cjdoris avatar Jan 26 '23 21:01 cjdoris

it still doesn't work, if I have a package that looks like this:

module WVZPythonExt
    function __init__()
    ▏   if contains(ENV["PYTHONHOME"], "cern.ch/lcg/release")
    ▏   @info "Using LCG Python, if it doesn't work try delete `.CondaPkg/` folder"
    ▏   ▏   ▏   # use LCG release python
    ▏   ▏   ▏   ENV["JULIA_CONDAPKG_BACKEND"] = "Null"
    ▏   end
    end

    using PythonCall, FHist, Serialization

    export serial_to_root

    function make_TH1D(h)
    ▏   np = pyimport("numpy")
    ▏   pyhist = pyimport("hist")

when I using this pkg, PythonCall would just try to download conda and does its own thing

Moelf avatar Jun 13 '23 21:06 Moelf

The __init__() function is not called until after the module is loaded (and PythonCall is already initialised) which is too late. You need to set the env var before loading the package.

cjdoris avatar Jun 15 '23 08:06 cjdoris

having it outside of __init__() and before using PythonCall also doesn't work

Moelf avatar Jun 15 '23 12:06 Moelf

if I have this:

module WVZPythonExt
    ENV["PYTHONPATH"] = readchomp(`which python3`)
    @show ENV["PYTHONPATH"]
    ENV["JULIA_CONDAPKG_BACKEND"] = "Null"
    using PythonCall, FHist, Serialization

I expect to not need local Python, but right now

julia> using WVZPythonExt
[ Info: Precompiling WVZPythonExt [473e61d0-be27-49e9-a959-28cb081f953f]
ENV["PYTHONPATH"] = "/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Thu/x86_64-centos7-gcc11-opt/bin/python3"
ERROR: InitError: Python executable "/home/jiling/.julia/dev/WVZAnalysis/.CondaPkg/env/bin/python" is not executable.

Moelf avatar Jun 15 '23 17:06 Moelf

basically I'm looking for:

  • use external Python 100%
  • or use internal Python 100%

neither seems doable programmatically without user starting Julia with specific ENV

Moelf avatar Jun 15 '23 17:06 Moelf

When I said "you need to set the env var before loading the package" I meant before loading the WVZPythonExt package. You can't reliably do this from within a package which itself depends on PythonCall because PythonCall will be initialised before your package can do anything.

So you need to do something like:

ENV["JULIA_CONDAPKG_BACKEND"] = "Null"
using WVZPythonExt

cjdoris avatar Jun 16 '23 15:06 cjdoris

yeah that always work but that require package users to start Julia in some special way.

What about this idea, maybe we can use Preference.jl and let PythonCall do this at compile time (surely if CUDA.jl can change driver by hooking up with Preference.jl, we can disable outside Python at the current step

Moelf avatar Jun 16 '23 17:06 Moelf

Yep, having PythonCall configurable with Preferences is on the to-do list.

I don't see why a package should be setting the CondaPkg backend though? Surely this depends on the user system and is therefore user config? What if a user of your package would rather have stuff automatically installed?

cjdoris avatar Jun 22 '23 16:06 cjdoris

That would be fine but that doesn't work with the HPC package environment our field uses, like, that's why I was asking how to completely disable outside python, because I want to be reproducible, I don't mind waste more disk space to install a completely separate environment

Moelf avatar Jun 22 '23 17:06 Moelf

This issue has been marked as stale because it has been open for 30 days with no activity. If the issue is still relevant then please leave a comment, or else it will be closed in 7 days.

github-actions[bot] avatar Sep 07 '23 01:09 github-actions[bot]

This issue has been closed because it has been stale for 7 days. If it is still relevant, please re-open it.

github-actions[bot] avatar Sep 15 '23 01:09 github-actions[bot]

Thanks

Moelf avatar Sep 22 '23 19:09 Moelf