[webgl] allow direct WebGL tensor creation/retrieval using texture handle

[pyu10055]

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template

System information

• TensorFlow.js version (you are using): 2.4.0
• Are you willing to contribute it (Yes/No):

Describe the feature and the current behavior/state. For ML models that involve media processing (i.e. bodypix model), there are couple places where expensive data copying could happen:

1. Creating tensor from result of video processing pipeline. Although TFJS has fromPixels API, but it requires a html element (video/image), for data that is processed on GPU it will need some work-around (download to cpu or create html element)
2. tensors from the ML execution can not be executed on GPU after the inference, unless using TFJS op APIs. If you have post-processing steps after the ML model execution that need to be executed on GPU, you will end up downloading the tensors and re-upload again.

Proposing two new tensor APIs to solve above problems, these APIs target tensors that are located on WebGL backend:

1. create tensor using WebGL texture handle, this allows data from GPU pre-processing steps to be converted to tensor.
2. retrieve texture handle for tensor, this would prevent downloading the tensor from GPU and enable continue post-processing on GPU.

Will this change the current api? How? This would not change any of the current APIs, and the proposed APIs shall throw exception for tensors that are not on WebGL backend.

Who will benefit with this feature? Media processing pipeline that utilize TFJS for model execution with WebGL acceleration. Any Other info.

[pmer]

This would be really great!! :)

You might need to be able to share a WebGL context with the user's application. Not sure if this would be a new API function or wrapped into something else. The user will probably know which canvas or OffscreenContext will want to use TFJS data, and could pass a reference to TFJS. Just a thought...

Previous discussion @ https://github.com/tensorflow/tfjs/issues/1607.

[sebavan]

Would also need a similar setup for WebGPU in the future :-)

[mmetelskiy]

Seems that #4376 was reverted by #4675 and this issue should be reopened

[NataliaDSmirnova]

Hi guys! Are there any updates about the feature? Does any other way to create a tensor from texture exist? The feature would be very useful for those who utilizes TFJS with any other WebGL-based lib in one WebGL context because it removes a necessity of redundant CPU/GPU sync.

[ats05]

The dataToGPU works as expected. Contrary to this, we expect additional methods to be added to create Tensors from GPU data.

[pyu10055]

we are in the process to unify the texture to tensor API across webgl and webgpu, please stay tuned.

[pyu10055]

@qjia7

[NataliaDSmirnova]

we are in the process to unify the texture to tensor API across webgl and webgpu, please stay tuned.

Thanks a million for your answer. I have a hope now 😃