Using PyJulia from Streamlit App

[mercicle]

Summary

I'm trying to use a julia function from a streamlit app using PyJulia. Created a toy example to test the interaction, simply returning a matrix from a julia functions based on a single parameter to specify the value of the diagonal elements.

Will also note at the outset that both julia_import_method = "api_compiled_false" and julia_import_method = "main_include" works when importing the function in Spyder IDE (rather than at the command line to launch the streamlit app via streamlit run streamlit_julia_test.py).

My project directory looks like:

─ my_project_directory

• julia_test.jl
• streamlit_julia_test.py

The julia function is in julia_test.jl and just simply returns a matrix with diagonals specified by the v parameter:

function get_matrix_from_julia(v::Int)
   m=[v 0
      0 v]
 return m
end

The streamlit app is streamlit_julia_test.py and is defined as:

import os
from io import BytesIO

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import streamlit as st

# options:
# main_include
# api_compiled_false
# dont_import_julia
julia_import_method = "api_compiled_false"

if julia_import_method == "main_include":

    import julia
    from julia import Main
    Main.include("julia_test.jl")

elif julia_import_method == "api_compiled_false":

    # works in Spyder IDE
    from julia.api import Julia
    jl = Julia(compiled_modules=False)

    this_dir = os.getcwd()
    julia_test_path = """include(\""""+ this_dir + """/julia_test.jl\"""" +")"""
    print(julia_test_path)
    jl.eval(julia_test_path)
    get_matrix_from_julia = jl.eval("get_matrix_from_julia")

elif julia_import_method == "dont_import_julia":
    print("Not importing ")
else:
    ValueError("Not handling this case:" + julia_import_method)


st.header('Using Julia in Streamlit App Example')

st.text("Using Method:" + julia_import_method)

matrix_element = st.selectbox('Set Matrix Diagonal to:', [1,2,3])

matrix_numpy = np.array([[matrix_element,0],[0,matrix_element]])

col1, col2 = st.columns([4,4])

with col1:

    fig, ax = plt.subplots(figsize=(5,5))
    sns.heatmap(matrix_numpy, ax = ax, cmap="Blues",annot=True)
    ax.set_title('Matrix Using Python Numpy')
    buf = BytesIO()
    fig.savefig(buf, format="png")
    st.image(buf)

with col2:

    if julia_import_method == "dont_import_julia":
        matrix_julia = matrix_numpy
    else:
        matrix_julia = get_matrix_from_julia(matrix_element)

    fig, ax = plt.subplots(figsize=(5,5))
    sns.heatmap(matrix_julia, ax = ax, cmap="Blues",annot=True)
    ax.set_title('Matrix from External Julia Script')
    buf = BytesIO()
    fig.savefig(buf, format="png")
    st.image(buf)

If the app were working correctly, it would look like this (which can be reproduced by setting the julia_import_method = "dont_import_julia" on line 13):

Testing

When I try julia_import_method = "main_include", I get the well known error:

julia.core.UnsupportedPythonError: It seems your Julia and PyJulia setup are not supported.

Julia executable:
    julia
Python interpreter and libpython used by PyCall.jl:
    /Users/myusername/opt/anaconda3/bin/python3
    /Users/myusername/opt/anaconda3/lib/libpython3.9.dylib
Python interpreter used to import PyJulia and its libpython.
    /Users/myusername/opt/anaconda3/bin/python
    /Users/myusername/opt/anaconda3/lib/libpython3.9.dylib

Your Python interpreter "/Users/myusername/opt/anaconda3/bin/python"
is statically linked to libpython.  Currently, PyJulia does not fully
support such Python interpreter.

The easiest workaround is to pass `compiled_modules=False` to `Julia`
constructor.  To do so, first *reboot* your Python REPL (if this happened
inside an interactive session) and then evaluate:

    >>> from julia.api import Julia
    >>> jl = Julia(compiled_modules=False)

Another workaround is to run your Python script with `python-jl`
command bundled in PyJulia.  You can simply do:

    $ python-jl PATH/TO/YOUR/SCRIPT.py

See `python-jl --help` for more information.

For more information, see:

    https://pyjulia.readthedocs.io/en/latest/troubleshooting.html

As suggested, when I set julia_import_method = "api_compiled_false", I get a seg fault:

include("my_project_directory/julia_test.jl")
2022-04-03 10:23:13.406 Traceback (most recent call last):
  File "/Users/myusername/opt/anaconda3/lib/python3.9/site-packages/streamlit/script_runner.py", line 430, in _run_script
    exec(code, module.__dict__)
  File "my_project_directory/streamlit_julia_test.py", line 25, in <module>
    jl = Julia(compiled_modules=False)
  File "/Users/myusername/.local/lib/python3.9/site-packages/julia/core.py", line 502, in __init__
    if not self.api.was_initialized:  # = jl_is_initialized()
  File "/Users/myusername/.local/lib/python3.9/site-packages/julia/libjulia.py", line 114, in __getattr__
    return getattr(self.libjulia, name)
  File "/Users/myusername/opt/anaconda3/lib/python3.9/ctypes/__init__.py", line 395, in __getattr__
    func = self.__getitem__(name)
  File "/Users/myuserame/opt/anaconda3/lib/python3.9/ctypes/__init__.py", line 400, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x21b8e5840, was_initialized): symbol not found


signal (11): Segmentation fault: 11
in expression starting at none:0
Allocations: 35278345 (Pool: 35267101; Big: 11244); GC: 36
zsh: segmentation fault  streamlit run streamlit_julia_test.py

I've also tried the alternative recommendation provided in the PyJulia response message regarding the use of:

python-jl my_project_directory/streamlit_julia_test.py

But I get this error when running the python-jl command:

INTEL MKL ERROR: dlopen(/Users/myusername/opt/anaconda3/lib/libmkl_intel_thread.1.dylib, 0x0009): Library not loaded: @rpath/libiomp5.dylib
  Referenced from: /Users/myusername/opt/anaconda3/lib/libmkl_intel_thread.1.dylib
  Reason: tried: '/Applications/Julia-1.7.app/Contents/Resources/julia/bin/../lib/libiomp5.dylib' (no such file), '/usr/local/lib/libiomp5.dylib' (no such file), '/usr/lib/libiomp5.dylib' (no such file).
Intel MKL FATAL ERROR: Cannot load libmkl_intel_thread.1.dylib.

So I'm stuck, thanks in advance for a modified reproducible example or instructions for the following system specs!

System specs:
Mac OS Monterey 12.2.1 (Chip - Mac M1 Pro)
Python 3.9.7
Julia 1.7.2
PyJulia 0.5.8.dev
Streamlit 1.7.0

[mercicle]

Hi @tkf , would you be able to take a look at this?

[MilesCranmer]

+1

For the record I seem to be able to get it working with streamlit once (without conda), but I see a segmentation fault when it tries to re-run the script. @tkf is anybody maintaining PyJulia that we can reach out to?