gradio Extend Interface.from_pipeline() to support Transformers.js.py pipelines on Lite

Description

Closes: #8033

To make something like this possible on Lite:

import gradio as gr
from transformers_js_py import import_transformers_js

transformers = await import_transformers_js()
pipeline = transformers.pipeline

pipe = await pipeline('object-detection')

demo = gr.Interface.from_pipeline(pipe)

demo.launch()

Apr 17 '24 17:04 whitphx

🪼 branch checks and previews

•	Name	Status	URL
	Spaces	ready!	Spaces preview
	Website	ready!	Website preview
	Storybook	ready!	Storybook preview
:unicorn:	Changes	detecting...

Install Gradio from this PR

pip install https://gradio-builds.s3.amazonaws.com/b785b41a31d899665235d23fb87bda53a72dac79/gradio-4.28.3-py3-none-any.whl

Install Gradio Python Client from this PR

pip install "gradio-client @ git+https://github.com/gradio-app/gradio@b785b41a31d899665235d23fb87bda53a72dac79#subdirectory=client/python"

Apr 17 '24 17:04 gradio-pr-bot

🦄 change detected

This Pull Request includes changes to the following packages.

Package	Version
`@gradio/lite`	`minor`
`gradio`	`minor`

[ ] Maintainers can select this checkbox to manually select packages to update.

With the following changelog entry.

Extend Interface.from_pipeline() to support Transformers.js.py pipelines on Lite

Maintainers or the PR author can modify the PR title to modify this entry.

Something isn't right?

Maintainers can change the version label to modify the version bump.
If the bot has failed to detect any changes, or if this pull request needs to update multiple packages to different versions or requires a more comprehensive changelog entry, maintainers can update the changelog file directly.

Apr 17 '24 17:04 gradio-pr-bot

Is render=False necessary?
Should examples be provided?

Apr 17 '24 18:04 whitphx

Very exciting @whitphx! cc @xenova for visibility 🔥

Apr 18 '24 19:04 abidlabs

Should add the examples as well? It looks too much and harder to maintain, but it contributes to user's convenience.

Apr 23 '24 08:04 whitphx

@whitphx I'm having some issues when I try to test this.

I checked out this branch and ran bash scripts/build_lite.sh
Then I wrote the following:

<!doctype html>
<!-- A demo HTML file to test the bundled JS and CSS files -->
<html>
	<head>
		<meta charset="utf-8" />
		<meta
			name="viewport"
			content="width=device-width, initial-scale=1, shrink-to-fit=no, maximum-scale=1"
		/>

		<link rel="preconnect" href="https://fonts.googleapis.com" />
		<link
			rel="preconnect"
			href="https://fonts.gstatic.com"
			crossorigin="anonymous"
		/>
		<script type="module" crossorigin src="./dist/lite.js"></script>
		<link rel="stylesheet" href="./dist/lite.css" />
	</head>

	<body style="padding: 10px; height: 100%; width: 100%">
		<h1>Lorem Ipsum Dolor</h1>

		<gradio-lite>

			<gradio-requirements>
			transformers_js_py
			</gradio-requirements>
			
			<gradio-file name="app.py" entrypoint>
				import gradio as gr
				from transformers_js_py import import_transformers_js
				
				transformers = await import_transformers_js()
				pipeline = transformers.pipeline
				
				pipe = await pipeline('object-detection')
				
				demo = gr.Interface.from_pipeline(pipe)
				
				demo.launch()				
			</gradio-file>
			
		</gradio-lite>	

	</body>
</html>

And ran it using python -m http.server

When I run this, the app does not load, instead I see this error:

Uncaught (in promise) PythonError: Traceback (most recent call last):
  File "/lib/python3.11/site-packages/gradio/wasm_utils.py", line 55, in get_registered_app
    return app_map[app_id]
           ~~~~~~~^^^^^^^^
KeyError: 'JNqvwI3u'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<exec>", line 29, in _call_asgi_app_from_js
  File "/lib/python3.11/site-packages/gradio/wasm_utils.py", line 57, in get_registered_app
    raise GradioAppNotFoundError(
gradio.wasm_utils.GradioAppNotFoundError: Gradio app not found (ID: JNqvwI3u). Forgot to call demo.launch()?

even though I do have demo.launch()

Are you able to repro this?

Apr 24 '24 20:04 abidlabs

@abidlabs It's the problem #8067 solved. Now I merged the fix to this PR so it should work(, while you have to delete the indent inside the <gradio-file> tag as the dedent feature is not working on the file tags which should be fixed as a seaprate PR though -> #8126). Plz try it again, thanks.

Apr 25 '24 06:04 whitphx

Very cool, its working for me now @whitphx! I noticed a few issues I was testing:

(1) Could we display the name of the model that is being used instead of the task? For example, when I supply object-detection, I see in my gradio-lite app:

Whereas if if I do from_pipeline() in regular gradio, I see the model name:

(2) I tried the automatic-speech-recognition pipeline, but I got two errors:

A) With file upload, the app errors out:

and the console shows this:

B) If I click the microphone, I get these errors:

This is a very cool integration and we should make sure to document this! We could add to our existing guides/08_gradio-clients-and-lite/gradio-lite.md guide or even better, create a new Guide focused on integrations between gradio/lite and transformer.js where we describe the transformers_js_py library and go into more detail about supported pipelines

Re: examples, they are nice to have, but I don't think necessary. If you'd like to add them, you can probably copy most of the ones from this file: gradio/external.py

Apr 25 '24 21:04 abidlabs

I went through and tested a bunch of the pipelines:

fill-mask ✅
text-classification ✅
question-answering ✅
zero-shot-classification ✅

The pipelines that had issues:

feature-extraction -- noticed an issue where the numerical values were displayed as a single comma-separated string instead of being spread out over multiple columns
depth-estimation - works although I see this error in the console:

audio-classification - same error as automatic-speech-recognition above

As a side note, I just see this screen (because I think it takes a long time for the model to load). I wonder if we could provide a more informative message?

Apr 25 '24 21:04 abidlabs

@abidlabs Thank you for the review!

(1) Could we display the name of the model that is being used instead of the task?

Will take a look 👍

(2) I tried the automatic-speech-recognition pipeline, but I got two errors:

A) With file upload, the app errors out: and the console shows this:

As the error says, you need to install numpy and scipy by adding them to the <gradio-requirements> element as below.

<gradio-requirements>
transformers_js_py
scipy
numpy
</gradio-requirements>

B) If I click the microphone, I get these errors:

Actually I couldn't reproduce it in my env but no microphones are found in my case. Will check it.

About your error, something like that happens when you opens the remote page via http:// protocol (not HTTPS) due to the spec described in this page as below.

This can also happen if you try to call getUserMedia() in an insecure context, since navigator.mediaDevices is undefined in an insecure context.

Does this help?

we should make sure to document this!

Yes, I will 👍

feature-extraction -- noticed an issue where the numerical values were displayed as a single comma-separated string instead of being spread out over multiple columns

Thanks, will take a look.

depth-estimation - works although I see this error in the console:

This 404 error in the dev console is expected in the Wasm mode and it's correct behavior. The app first tries to load the image via a normal HTTP access, then fallback to loading it from the Wasm worker after it fails due to 404.

As a side note, I just see this screen (because I think it takes a long time for the model to load). I wonder if we could provide a more informative message?

Thanks, will take a look.

Apr 26 '24 05:04 whitphx

As the error says, you need to install numpy and scipy by adding them to the element as below.

But numpy is already a dependency of gradio so why does it need to be installed? And why do we need scipy if its not used in the gradio package?

Apr 27 '24 11:04 abidlabs

But numpy is already a dependency of gradio so why does it need to be installed? And why do we need scipy if its not used in the gradio package?

It's used in transformers.js.py to deal with the audio (wav) file.
- https://github.com/whitphx/transformers.js.py/blob/main/transformers_js_py/audio.py
- More detail: Transformers.js uses JS's AudioContext for this purpose in a browser's renderer process, but it can't be used in a WebWorker process. So transformers.js.py uses scipy instead as a more general audio-handling method.
This error message is shown when at least one of numpy and scipe is not found from transformers.js.py. So actually you don't have to add numpy explicitly in the case of Gradio-Lite. Adding it is just for clearer dependency declaration.

Apr 29 '24 03:04 whitphx

I don’t understand — if numpy and scipy are needed by transformers_js_py, then they should be dependencies, and then one should not get an error if they are not installed?

unrelated but it would be great to include internal link this updated syntax in the docs for this!

Apr 29 '24 05:04 abidlabs

scipy is only needed by the audio reader and I don't want to install it in all env.

unrelated but it would be great to include internal link this updated syntax in the docs for this!

👍

Apr 29 '24 07:04 whitphx

Ah, it should be included as an extra dep like transformers_js_py[audio]. Will do it.

Apr 29 '24 07:04 whitphx

Thanks @whitphx for clarifying! Let me know when you'd like another review

Apr 29 '24 14:04 abidlabs

Now this code should work.

import gradio as gr
from transformers_js_py import pipeline

pipe = await pipeline("feature-extraction")

demo = gr.Interface.from_pipeline(pipe)

if __name__ == "__main__":
    demo.launch()

For audio pipelines, install transformers_js_py[audio].

May 02 '24 05:05 whitphx

Does transformers_js_py.pipeline support loading specific models like transformers.js pipeline?

I ran this code:

import gradio as gr
from transformers_js_py import pipeline

pipe = await pipeline('sentiment-analysis', 'Xenova/bert-base-multilingual-uncased-sentiment');

demo = gr.Interface.from_pipeline(pipe)

if __name__ == "__main__":
    demo.launch()

and the loaded model says its still "bert" (the default). I'm not sure if the model is mislabeled or the the wrong model was loaded

May 02 '24 05:05 abidlabs

This is what I see when I run the code above:

May 02 '24 05:05 abidlabs

@abidlabs

Does transformers_js_py.pipeline support loading specific models like transformers.js pipeline?

Yes.

And the demo title is taken from pipeline.model.config.model_type and it's actually "bert" in that case. What do you think should be printed as a title?

fyi, the available attrs in pipe.model is like this (print(pipe.model)):

<TjsProxy({'main_input_name': 'input_ids', 'config': <TjsProxy({'model_type': 'bert', 'is_encoder_decoder': False, '_name_or_path': 'nlptown/bert-base-multilingual-uncased-sentiment', '_num_labels': 5, 'architectures': ['BertForSequenceClassification'], 'attention_probs_dropout_prob': 0.1, 'classifier_dropout': None, 'directionality': 'bidi', 'finetuning_task': 'sentiment-analysis', 'hidden_act': 'gelu', 'hidden_dropout_prob': 0.1, 'hidden_size': 768, 'id2label': {'0': '1 star', '1': '2 stars', '2': '3 stars', '3': '4 stars', '4': '5 stars'}, 'initializer_range': 0.02, 'intermediate_size': 3072, 'label2id': {'1 star': 0, '2 stars': 1, '3 stars': 2, '4 stars': 3, '5 stars': 4}, 'layer_norm_eps': 1e-12, 'max_position_embeddings': 512, 'num_attention_heads': 12, 'num_hidden_layers': 12, 'output_past': True, 'pad_token_id': 0, 'pooler_fc_size': 768, 'pooler_num_attention_heads': 12, 'pooler_num_fc_layers': 3, 'pooler_size_per_head': 128, 'pooler_type': 'first_token_transform', 'position_embedding_type': 'absolute', 'transformers_version': '4.29.2', 'type_vocab_size': 2, 'use_cache': True, 'vocab_size': 105879})>, 'session': <TjsProxy({'handler': <TjsProxy({'sessionId': 176556624, 'inputNames': ['input_ids', 'attention_mask', 'token_type_ids'], 'outputNames': ['logits']})>})>, 'can_generate': False, '_runBeam': None, '_getStartBeams': None, '_updateBeam': None, '_forward': <TjsProxy({})>})>

May 02 '24 05:05 whitphx

Ah I realize there's some existing discrepancy which caused me some confusion.

So if you do:

import gradio as gr

gr.load("deepset/roberta-base-squad2", src="models").launch()

it prints the name of the model as the title:

However, if you load from the pipeline, it just prints the model type:

import gradio as gr
from transformers import pipeline

pipe = pipeline(model="deepset/roberta-base-squad2")
gr.Interface.from_pipeline(pipe).launch()

Between these, I think its clearest for developers if the model name deepset/roberta-base-squad2 appears. I'd suggest that we make that change for both gr.Interface.from_pipeline and for the Gradio-Lite equivalent.

May 02 '24 06:05 abidlabs

Other than that nit & the docs, everything else LGTM @whitphx!

May 02 '24 06:05 abidlabs

@abidlabs

I think its clearest for developers if the model name deepset/roberta-base-squad2 appears.

I see. It can be taken from pipe.model.config._name_or_path. Don't you think the task name is necessary?

May 02 '24 07:05 whitphx

Don't you think the task name is necessary?

No strong opinion, I'm leaning towards not necessary. Happy to take a final pass once the docs are in!

May 02 '24 19:05 abidlabs

Made a few small tweaks to the docs e.g. to add mention of transformers_js_py[audio] but otherwise great PR @whitphx!

May 03 '24 07:05 abidlabs

gradio gradio copied to clipboard

Extend Interface.from_pipeline() to support Transformers.js.py pipelines on Lite

Description

🪼 branch checks and previews

🦄 change detected

This Pull Request includes changes to the following packages.

With the following changelog entry.

Something isn't right?

gradio
gradio copied to clipboard