smart dep_manager

[dcolinmorgan]

smart dep_manager without cu_cat stuff

[lmeyerov]

Note conflicts, update from latest head?

[lmeyerov]

@dcolinmorgan still conflicts w main

[dcolinmorgan]

@dcolinmorgan still conflicts w main

• [x] merged master and passing
• [x] colab cpu
• [x] colab GPU
• [x] colab GPU + RAPIDS

[lmeyerov]

I was just trying on colab, and still a weird experience on this PR, here is CPU/umap_learn:

! pip install -q graphistry[umap_learn]
! pip install -q git+https://github.com/graphistry/pygraphistry.git@dev/depman #[umap_learn]
import pandas as pd
df = pd.read_csv('https://raw.githubusercontent.com/graphistry/pygraphistry/master/demos/data/honeypot.csv')
df.shape, df.dtypes
import graphistry
graphistry.__version__

=> 0.34.3+446.g6f455d1

and then the test:

graphistry.nodes(df).umap(engine='umap_learn')
1

=>

WARNING:graphistry.utils.lazy_import:Unexpected exn during lazy import
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/cupy/__init__.py", line 17, in <module>
    from cupy import _core  # NOQA
  File "/usr/local/lib/python3.10/dist-packages/cupy/_core/__init__.py", line 3, in <module>
    from cupy._core import core  # NOQA
ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/graphistry/utils/lazy_import.py", line 11, in lazy_cudf_import
    import cudf  # type: ignore
  File "/usr/local/lib/python3.10/dist-packages/cudf/__init__.py", line 12, in <module>
    import cupy
  File "/usr/local/lib/python3.10/dist-packages/cupy/__init__.py", line 19, in <module>
    raise ImportError(f'''
ImportError: 
================================================================
Failed to import CuPy.

If you installed CuPy via wheels (cupy-cudaXXX or cupy-rocm-X-X), make sure that the package matches with the version of CUDA or ROCm installed.

On Linux, you may need to set LD_LIBRARY_PATH environment variable depending on how you installed CUDA/ROCm.
On Windows, try setting CUDA_PATH environment variable.

Check the Installation Guide for details:
  https://docs.cupy.dev/en/latest/install.html

Original error:
  ImportError: libcuda.so.1: cannot open shared object file: No such file or directory
================================================================
ERROR:graphistry.utils.dep_manager:cudf installed but misconfiguredERROR:graphistry.utils.dep_manager:cudf installed but misconfiguredERROR:graphistry.utils.dep_manager:cudf installed but misconfiguredERROR:graphistry.utils.dep_manager:cudf installed but misconfiguredERROR:graphistry.utils.dep_manager:cudf installed but misconfiguredWARNING:graphistry.umap_utils:* Ignoring target column of shape (220, 0) in UMAP fit, as it is not one dimensional
1

[dcolinmorgan]

had to replace a few dangling lazy_cudf imports

is the plan to swap other lazy imports before / after merge? pretty easy, one just need be thorough

[lmeyerov]

howdy!

main has gone forward a bunch since this, see conflicts

i think it's easier and preferable to land this first. i've been more in this repo recently on multiple gpu systems, so easier for me to test now