Refactor: split out routines_creation and compile with cache from core.pyx (as marked as TODO)

[qingfengxia]

This is part of trial of porting cupy to ascend. for this MR, it is completed, I just split the core.pyx without touching any code logics, reorder some import.

My future trial will not seek to merge within half year, this MR will let my life easier for the future merge.

compiling is fine, using stub backend. I may need help to complete this refactor work, lack of the testing hardware. even the readthedocs failed, no idea how I can fix.

can someone apply my patch ?

[mergify[bot]]

This pull request is now in conflicts. Could you fix it @qingfengxia? 🙏

[qingfengxia]

testing with one gpu, it seems core/init.py should be fixed after split

from cupy._core.core import _internal_ascontiguousarray  # NOQA
from cupy._core.core import _internal_asfortranarray  # NOQA
from cupy._core.core import array  # NOQA
from cupy._core.core import ascontiguousarray  # NOQA
from cupy._core.core import asfortranarray  # NOQA
from cupy._core.core import divmod  # NOQA
from cupy._core.core import elementwise_copy  # NOQA
from cupy._core.core import ndarray  # NOQA

[seberg]

reorder some import.

I would think just an import cupy should also trigger the same issue locally for you (it may depend on the Python version, though)? You can easily get the results e.g. from this run too (see also details).

That error suggests the import of cupy/cuda/compiler.py goes in circles. It would be curious to figure out why this refactor triggers it, but moving some of the functions there into a different module like a new _util (to allow importing them from there without going in circles) may also be an approach.

File "/home/docs/checkouts/readthedocs.org/user_builds/cupy/envs/9408/lib/python3.12/site-packages/cupy/cuda/compiler.py", line 16, in <module>
	    from cupy.cuda import function
File "cupy/cuda/function.pyx", line 1, in init cupy.cuda.function
	  File "cupy/cuda/texture.pyx", line 1, in init cupy.cuda.texture
	  File "cupy/_core/_routines_creation.pyx", line 1, in init cupy._core._routines_creation
	  File "cupy/_core/_routines_manipulation.pyx", line 837, in init cupy._core._routines_manipulation
	  File "cupy/_core/_kernel.pyx", line 789, in cupy._core._kernel.ElementwiseKernel.__init__
	AttributeError: partially initialized module 'cupy.cuda.compiler' has no attribute 'is_valid_kernel_name' (most likely due to a circular import)

[qingfengxia]

yes, I also just found out import in stub mode, can trigger some errors. I am quite new to cython.

Please help to explain, why and how core.ndarray is exported.

# after split out creation, some types must be exported
cdef object ndarray  # this declear does not work in toplevel  cupy.__init__.py
# I add this two line above, but this does not work, how can I export  this ndarray class? extract all the class def in core.pyx into core.pxd here, I wonder why, preivously,  ndarray is exported?  

cdef class _ndarray_base:

core.pyx , why there is no cdef or cpdef? it that a pure python, automatically can be import? while I split this ndarray class out as a pure python module, _ndarray.py , hope it can solve the problem

class ndarray(_ndarray_base):

this maybe the reason, still a simple split lead to lots of problem

reorder some import.

I would think just an import cupy should also trigger the same issue locally for you (it may depend on the Python version, though)? You can easily get the results e.g. from this run too (see also details).

That error suggests the import of cupy/cuda/compiler.py goes in circles. It would be curious to figure out why this refactor triggers it, but moving some of the functions there into a different module like a new _util (to allow importing them from there without going in circles) may also be an approach.

File "/home/docs/checkouts/readthedocs.org/user_builds/cupy/envs/9408/lib/python3.12/site-packages/cupy/cuda/compiler.py", line 16, in <module>
	    from cupy.cuda import function
File "cupy/cuda/function.pyx", line 1, in init cupy.cuda.function
	  File "cupy/cuda/texture.pyx", line 1, in init cupy.cuda.texture
	  File "cupy/_core/_routines_creation.pyx", line 1, in init cupy._core._routines_creation
	  File "cupy/_core/_routines_manipulation.pyx", line 837, in init cupy._core._routines_manipulation
	  File "cupy/_core/_kernel.pyx", line 789, in cupy._core._kernel.ElementwiseKernel.__init__
	AttributeError: partially initialized module 'cupy.cuda.compiler' has no attribute 'is_valid_kernel_name' (most likely due to a circular import)

[qingfengxia]

There is circular import _core/_routines_creation -> core.pyx -> cuda/function.pyx -> cuda/texture.pyx -> _core/_routines_creation

[mergify[bot]]

This pull request is now in conflicts. Could you fix it @qingfengxia? 🙏