JS function executed via the `js` option with binary data should be passed blob data too

[whitphx]

Gradio can execute JS function via the js option. When the input is binary data like an image, we usually want the data in JS, but currently a FileData object like below is passed and it doesn't have a blob field value.

In normal Gradio, we can get the data by fetching the URL like fetch(inp.url), but it's less convenient and this way is not available on Lite, so the blob value should be set by default.

{
  alt_text: undefined,
  blob: undefined,
  is_stream: undefined,
  meta: {_type: 'gradio.FileData'},
  mime_type: "image/webp",
  orig_name: "5eabff9b8736638402a2190ef9f2240a.webp",
  path: "/private/var/folders/1p/xkgpb5r96x7g63qty32s3p4r0000gn/T/gradio/206225546c14c14cf7c50358c0e806fa38557a12/5eabff9b8736638402a2190ef9f2240a.webp",
  size: 12758,
  url: "http://127.0.0.1:7860/file=/private/var/folders/1p/xkgpb5r96x7g63qty32s3p4r0000gn/T/gradio/206225546c14c14cf7c50358c0e806fa38557a12/5eabff9b8736638402a2190ef9f2240a.webp",
}

import gradio as gr

js = """
function processImage(inp) {
    console.log(inp)
    return inp
}
"""

with gr.Blocks(js=js) as demo:
    inp = gr.Image(type="numpy")
    out = gr.Image()
    submit = gr.Button("Submit")
    submit.click(None, inp, out, js=js)

demo.launch()

Ref: https://huggingface.slack.com/archives/C055NMB0S87/p1712807154553579?thread_ts=1712708043.587169&cid=C055NMB0S87

---

Some considerations:

• Blob encoding/passing should not be enabled by default because it introduces extra footprint and Blob is not always availalbe.
  • Should it be implemented for Lite, or for both and turned on/off by some flag?

[whitphx]

In the first place, the blob field is not defined in FileData on the Python side. https://github.com/gradio-app/gradio/blob/2a5cb978240960fd8b150a1eae6ae8328a5c0c59/gradio/data_classes.py#L173-L180 In the example above, the blob field is created (but its value is not set) in the <Upload /> component on the JS side and just passed to the JS function. https://github.com/gradio-app/gradio/blob/2a5cb978240960fd8b150a1eae6ae8328a5c0c59/js/upload/src/Upload.svelte#L81

Supporting the blob field is a not intended design?

[freddyaboulton]

My thought is that the blob should only be enabled for Lite because the blob should not be sent to the server when an event is triggered.

[whitphx]

Thanks!

the blob should not be sent to the server when an event is triggered.

Agreed with that point. But I think it's not a reason to restrict it only for Lite because the data can be converted to blob only when passed to the user-defined JS func dispatched via the js option anyway, even in the case of normal Gradio? Sorry if I missed something.

[freddyaboulton]

I think it's that is tricky because each component can define their own data structure for the value and so the app would have to traverse the value and convert anything that's possibly a file to a blob. We've had troubles with that with the python client.

[whitphx]

Maybe we don't have to define each component's data structure but just to convert the file-like data into blob in get_data(): https://github.com/gradio-app/gradio/blob/576731051f4945057b32274374de9c835085c24c/js/app/src/init.ts#L216-L225 with something like this pseudo code:

const value = comp.props.value;
if (is_file_info(value)) {
    value.blob = // get the blob somehow
}
return value;

, but anyway this is true and it can be tricky as you wrote.

the app would have to traverse the value and convert anything that's possibly a file to a blob.

---

I'm thinking of another option exposing a special fetch-like method to the developer specifically in the Lite case.

The goal is to provide the developer the way to access the binary data in the JS world somehow.

[freddyaboulton]

Actually the first approach shouldn't be as tricky as I thought. I forgot we introduced this meta key to help identify FileData objects. Writing is_file_info should be possible, we just need to take care to handle nested objects.