godot-proposals icon indicating copy to clipboard operation
godot-proposals copied to clipboard

Adding a way to fill a camera feed from addons and external Android library

Open elmajime opened this issue 5 months ago • 1 comments

Describe the project you are working on

ARCore integration in the form of a plugin based on XRInterfaceExtension

Describe the problem or limitation you are having in your project

The camera feed being provided by ARCore using GL_TEXTURE_EXTERNAL_OES, we need a way to give it to Godot.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Add a camera feed function that initializes a buffer as GL_TEXTURE_EXTERNAL_OES on Android

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

The ARCore plugin needs to work as follows:

// Initilazing the feed in a way that it uses GL_TEXTURE_EXTERNAL_OES to create the buffer
feed->set_external(width, height); // open to name suggestion

// Giving the feed handle to ARCore, 
ArSession_setCameraTextureName(ar_session, feed->get_texture_tex_id(CameraServer::FEED_YCBCR_IMAGE));

This would in turn make use of something like :

RID TextureStorage::texture_set_external(RID p_texture, int p_width, int p_height) {
	Texture texture;
	texture.width = p_width;
	texture.height = p_height;
	texture.alloc_width = texture.width;
	texture.alloc_height = texture.height;
	texture.mipmaps = 1;
	texture.format = Image::FORMAT_RGB8;
	texture.type = Texture::TYPE_2D;
#ifdef ANDROID_ENABLED
	texture.target = GL_TEXTURE_EXTERNAL_OES;
#endif
	_get_gl_image_and_format(Ref<Image>(), texture.format, texture.real_format, texture.gl_format_cache, texture.gl_internal_format_cache, texture.gl_type_cache, texture.compressed, false);
	texture.total_data_size = Image::get_image_data_size(texture.width, texture.height, texture.format, texture.mipmaps);
	texture.active = true;
	glGenTextures(1, &texture.tex_id);
	GLES3::Utilities::get_singleton()->texture_allocated_data(texture.tex_id, texture.total_data_size, "Texture 2D");

	return texture_owner.make_rid(texture);
}

And would be drawn using something like this in RasterizerSceneGLES3:

if (draw_feed && camera_feed_id > -1) {
	RENDER_TIMESTAMP("Render Camera feed");

	glEnable(GL_DEPTH_TEST);
	glEnable(GL_BLEND);
	glEnable(GL_CULL_FACE);
	glCullFace(GL_BACK);

	Ref<CameraFeed> feed = CameraServer::get_singleton()->get_feed_by_id(camera_feed_id);

	if (feed.is_valid()) {
		RID camera_YCBCR = feed->get_texture(CameraServer::FEED_YCBCR_IMAGE);
		GLES3::TextureStorage::get_singleton()->texture_bind(camera_YCBCR, 0);

		GLES3::FeedEffects *feed_effects = GLES3::FeedEffects::get_singleton();
		feed_effects->draw_feed();

		glDisable(GL_BLEND);
	}
}

If this enhancement will not be used often, can it be worked around with a few lines of script?

This can't be done in script

Is there a reason why this should be core and not an add-on in the asset library?

This is the most important requirement for ARCore to be made available for Godot as a plugin

elmajime avatar Sep 07 '24 05:09 elmajime