FlowFuse
Asset Service: Support larger file uploads
Description
Per https://github.com/FlowFuse/node-red/issues/74#issuecomment-3285828399 our asset service currently only supports the uploading of files 10 MB or less. This is very restrictive for the value that can be attained by the Asset service.
In particular, for the full value of ONNX Nodes to be attained, we should be looking at supporting upwards of 500 MB files.
Which customers would this be available to
None
Have you provided an initial effort estimate for this issue?
I have provided an initial effort estimate
@ZJvandeWeg you mentioned that customer you were dealing with had files uploaded of 2 GB in size? Were they accumulatively appended to, rather than uploaded as single entities?
@ZJvandeWeg you mentioned that customer you were dealing with had files uploaded of 2 GB in size? Were they accumulatively appended to, rather than uploaded as single entities?
I don't know, I saw files well over 2GB each, about 20.
Adding for context: this work is needed so that AI-backed improvements we're making to Node-RED are able to function as intended.
Adding for context: this work is needed so that AI-backed improvements we're making to Node-RED are able to function as intended.
I'm not sure what you mean with this, please elaborate?
From a quick discussion yesterday with Nick, these were likely generated by the system itself and just written to disk. The extension of the files is also .core so I suspect it's not uploaded in the traditional sense.
Adding for context: this work is needed so that AI-backed improvements we're making to Node-RED are able to function as intended.
I'm not sure what you mean with this, please elaborate?
Per https://github.com/FlowFuse/node-red/issues/74#issuecomment-3285595128 for the ONNX nodes, it will not be possible for users to connect their own trained AI models with Node-RED without an expansion of our asset service.
Per https://github.com/FlowFuse/node-red/issues/74#issuecomment-3285595128 for the ONNX nodes, it will not be possible for users to connect their own trained AI models with Node-RED without an expansion of our asset service.
Noticed this isn't scheduled - should it be?
Needed - no*1, convenient yes.
Currently, to add your own model of a size larger than our upload limit is difficult, but not impossible (needs flows that permit an endpoint upload in chunks) - very clunky and somewhat specialised.
However, should a user wish to easily upload their own model (via FF UI assets page) - they cannot.
*1 Using any of the "named" hugging face models will auto download to the filestylem therefore bypassing the limit.
*1 Using any of the "named" hugging face models will auto download to the filestylem therefore bypassing the limit.
Thanks @Steve-Mcl , I'll take this out of the plan for now, until there's time for a more comprehensive look at assets.