godot-python icon indicating copy to clipboard operation
godot-python copied to clipboard

[Godot 4] Better numpy support for image / np.arrays

Open TheBricktop opened this issue 3 years ago • 2 comments

As @touilleMan himself wrote in one of the issues, there is a considerable amount of developers that are interested in using python in godot for image processing / computer vision and AI or machine learning. Now the communication between numpy standard library and Godots way of handling large byte arrays are not so optimal so there is a room for improvement. This issue is made to collect the ideas, discuss possibilities and usecases for numpy in godot.

TheBricktop avatar Sep 08 '22 20:09 TheBricktop

My input:

  • [ ] More performant communication from opencv / numpy.arrays to godot types
  • [ ] functions that encapsulate creating images or streaming audio files generated in python.

TheBricktop avatar Sep 08 '22 20:09 TheBricktop

I know you commented on my related issue for this (#305) but for the sake of completeness, this is how one can currently go about efficiently converting a Matplotlib canvas (a Numpy Array) to a Godot PoolByteArray:

def canvas_to_godot_image(self, canvas):
	image = Image()
	
	width = canvas.get_width_height()[1]
	height = canvas.get_width_height()[0]
	
	size = width * height * 3
	
	pool_array = PoolByteArray()
	pool_array.resize(size)
	
	with pool_array.raw_access() as ptr:
		numpy_array = np.ctypeslib.as_array(ctypes.cast(ptr.get_address(), ctypes.POINTER(ctypes.c_uint8)), (size,))
		numpy_array[:] = np.frombuffer(canvas.tostring_rgb(), dtype=np.uint8).reshape(canvas.get_width_height()[::-1] + (3,)).flatten()
	
	image.create_from_data(height, width, False, Image.FORMAT_RGB8, pool_array)
	return image

As you wrote, it would be great if godot-python could provide this kind of functionality out of the box.

I think there are two ways in which this could be done:

  1. Define a PoolByteArray constructor specifically to construct from a Numpy Array, or
  2. Provide a Numpy-specific util class with a function such as NumpyUtil.numpy_array_to_pool_byte_array(array)

I think the first one would be more comfortable for users since they don't need to know that Numpy Arrays are a special case with a performant conversion method. However, the second one would be better for the godot-python codebase since it doesn't pollute the core codebase with Numpy-specific functionality.

Alternatively (or in addition), I think both options are realizable with compile options specifying whether Numpy-specific functionality should be compiled in or not - that would be another way of keeping things a bit separate.

@touilleMan what are your thoughts?

kb173 avatar Nov 01 '22 13:11 kb173