codon

codon copied to clipboard

This is the simple python script:

` import numpy as np

the_array = np.array([49, 7, 44, 27, 13, 35, 71])

an_array = np.where(the_array > 30, 0, the_array) print(an_array) `

I can run it under annaconda virtual environment with python: python t_pkg.py [ 0 7 0 27 13 0 0]

But it can't be run it with codon: codon run t_pkg.py t_pkg.py:1:8-13: error: no module named 'numpy' t_pkg.py:3:13-50: error: no module named 'np' t_pkg.py:5:12-50: error: no module named 'np' t_pkg.py:6:7-15: error: name 'an_array' is not defined

How to fix this? Thanks!

You need two things:

• Make sure that CODON_PYTHON points to Anaconda Python release (if not set, Codon might end up using the system Python release and that is often not the desired behaviour)
• Make sure that PYTHONHOME points to Anaconda Python (or venv) package repository (otherwise, it will default to the system repository).

You need two things:

• Make sure that CODON_PYTHON points to Anaconda Python release (if not set, Codon might end up using the system Python release and that is often not the desired behaviour)
• Make sure that PYTHONHOME points to Anaconda Python (or venv) package repository (otherwise, it will default to the system repository).

So how to make it?

I used conda env list to get the environments' paths, so I got that

base * /opt/homebrew/Caskroom/miniforge/base

Then I ran the command

export CODON_PATH= /opt/homebrew/Caskroom/miniforge/base

However, when I ran the .py file with numpy called in it, the errors still occurred.

Same experience here, doesn't seem to work:

$ export CODON_PYTHON=/home/toaster/mambaforge-pypy3/envs/puma-lab/bin/python

# this results in the same error
# $ export CODON_PYTHON=/home/toaster/mambaforge-pypy3/lib/libpypy3.9-c.so

$ export PYTHONHOME=/home/toaster/mambaforge-pypy3/envs/puma-lab
$ condon run bin/script.py 
Python path configuration:
  PYTHONHOME = '/home/toaster/mambaforge-pypy3/envs/puma-lab'
  PYTHONPATH = (not set)
  program name = '/usr/bin/python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/home/toaster/mambaforge-pypy3/envs/puma-lab'
  sys.base_exec_prefix = '/home/toaster/mambaforge-pypy3/envs/puma-lab'
  sys.platlibdir = 'lib'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/home/toaster/mambaforge-pypy3/envs/puma-lab'
  sys.exec_prefix = '/home/toaster/mambaforge-pypy3/envs/puma-lab'
  sys.path = [
    '/home/toaster/mambaforge-pypy3/envs/puma-lab/lib/python310.zip',
    '/home/toaster/mambaforge-pypy3/envs/puma-lab/lib/python3.10',
    '/home/toaster/mambaforge-pypy3/envs/puma-lab/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fc3aa9a2000 (most recent call first):
  <no Python frame>

Any suggestions? Can't find anything in the docs.

It's quite a BIG ISSUE!

If codon even cannot support python packages and can work only with pure codon code, it would be quite limited for application.

Codon can use Python packages, but they need to be imported from python:

from python import numpy as np

the_array = np.array([49, 7, 44, 27, 13, 35, 71])

an_array = np.where(the_array > 30, 0, the_array)
print(an_array)

(prints [ 0 7 0 27 13 0 0])

To set it up, the CODON_PYTHON env var should point to the Python shared library. For example, for me (on Mac) it is:

/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/lib/libpython3.9.dylib

@jmakov it looks like the first CODON_PYTHON points to the executable and the second points to a PyPy-related library? Is there a libpython3.10.so in /home/toaster/mambaforge-pypy3/envs/puma-lab/lib/ that you can set CODON_PYTHON to?

@arshajii thanks for the quick response! puma-lab is just the name of my mamba Python3.9 environment (no idea why codon sees Python3.10 (though it is installed on the system, along with Python3.9, 3.10 and 3.11)):

# This is from `codon`'s output. None of this files exist.
sys.path = [
    '/home/toaster/mambaforge-pypy3/envs/puma-lab/lib/python310.zip',
    '/home/toaster/mambaforge-pypy3/envs/puma-lab/lib/python3.10',
    '/home/toaster/mambaforge-pypy3/envs/puma-lab/lib/python3.10/lib-dynload',
  ]

As in my comment, I have also tried this with the same result:

$ export CODON_PYTHON=/home/toaster/mambaforge-pypy3/lib/libpypy3.9-c.so
# path to my `mamba/conda` environment
$ export PYTHONHOME=/home/toaster/mambaforge-pypy3/envs/puma-lab

It works. Thank you!

I guess if libpython.dylib is used, it won't be accelerated by codon, right? Is there anyway to accelerate those python packages using codon?

Hi there, I have tried the above step of exporting CODON_PYTHON or PYTHONHOME. But when I execute the program <xxyy.codon>, I got the following unknown error:

Assert failed: cannot find 'str.new:dispatch' [xxyy.codon:11:5] Expression: val Source: /github/workspace/codon/parser/visitors/typecheck/access.cpp:40 Aborted

Noticed that I have changed the import block into "from python import ...". Any ideas or suggestions would be appreciated. Thank you.

Same experience here, doesn't seem to work:

$ export CODON_PYTHON=/home/toaster/mambaforge-pypy3/envs/puma-lab/bin/python

# this results in the same error
# $ export CODON_PYTHON=/home/toaster/mambaforge-pypy3/lib/libpypy3.9-c.so

$ export PYTHONHOME=/home/toaster/mambaforge-pypy3/envs/puma-lab
$ condon run bin/script.py 
Python path configuration:
  PYTHONHOME = '/home/toaster/mambaforge-pypy3/envs/puma-lab'
  PYTHONPATH = (not set)
  program name = '/usr/bin/python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/home/toaster/mambaforge-pypy3/envs/puma-lab'
  sys.base_exec_prefix = '/home/toaster/mambaforge-pypy3/envs/puma-lab'
  sys.platlibdir = 'lib'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/home/toaster/mambaforge-pypy3/envs/puma-lab'
  sys.exec_prefix = '/home/toaster/mambaforge-pypy3/envs/puma-lab'
  sys.path = [
    '/home/toaster/mambaforge-pypy3/envs/puma-lab/lib/python310.zip',
    '/home/toaster/mambaforge-pypy3/envs/puma-lab/lib/python3.10',
    '/home/toaster/mambaforge-pypy3/envs/puma-lab/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fc3aa9a2000 (most recent call first):
  <no Python frame>

Any suggestions? Can't find anything in the docs.

The same error happened when I tried to change the PYTHONHOME as well. It just crashed my anaconda venv and conda command.

Using conda here. I exported the following variables:

export PYTHONHOME=/opt/conda/envs/envname/
export PYTHONPATH=/opt/conda/envs/envname/bin/python3
export CODON_PYTHON=/opt/conda/envs/envname/lib/libpython3.10.so

I was able to compile the python code ( codon build -release programname.py) changing the imports from:

import transformers
import torch
import numpy
import time

to

from python import transformers
from python import torch
from python import numpy
from python import time
from python import builtins

If conda env is loaded (typically via bash/zsh), you just need to set up CODON_PYTHON. It needs to point to a Python library, not an executable! In my case, I did CODON_PYTHON=$HOME/miniconda3/lib/libpython3.so codon run test.codon (on macOS, you will need to point to dylib, not so).

PYTHONPATH is not needed. PYTHONHOME should not be changed unless you have a custom setup (Fatal Python error: init_fs_encoding error indicates that PYTHONHOME is incorrect; in my experience, there is no need to change that in a loaded Conda env).

Same holds for venvs etc. Just make sure to point to correct Python library: e.g., do not point to pyenv or system library if using Conda, or vice versa. This is by design: we allow users to use custom Python versions (via CODON_PYTHON) and custom environments (via PYTHONPATH) per-app basis.

Please let me know if this resolves the issue.

P.S. If you need to set PYTHONPATH, run python -c "import sys; print(sys.path)" with your Python of choice to figure its exact contents.