Flowise icon indicating copy to clipboard operation
Flowise copied to clipboard

Refactor for loops

Open burn2delete opened this issue 1 year ago • 3 comments

Refactor some of the for loops into maps, including other code simplifications

burn2delete avatar Jun 10 '23 19:06 burn2delete

Results: Document Loaders:

  1. Cheerio: Pass image

chungyau97 avatar Jun 12 '23 13:06 chungyau97

going to put this on hold first until we have less PRs

HenryHengZJ avatar Jun 14 '23 18:06 HenryHengZJ

@burn2delete I was just having a glance at the PR code - because so many (20) files were altered - and I wonder if there are some maps happening now in your PR where, in fact, you want to keep the sequential dependence the slower for loop may provide.

In particular, and I may be getting this wrong because I was just glancing at the code, but you seem to be using map when assembling previous chat messages at one point into a context to be provided again. It could be argued that the sequence may not matter, but I think the stronger argument may be that the sequence COULD matter whereas the speedup in those instances is marginal? What do you think? I'm all for unrolling and parallelising, but if one simply does a search for all 'for' loops this doesn't filter if some of them are implicitly handling dependent sequences which shouldn't be parallelised and I don't know if you applied such checking given you did so many updates?

I realise this PR is almost a year old, so maybe things have overtaken this, but in case it matters for your own fork or if indeed this PR may yet get incorporated, I thought I should raise the issue before any merging.

deep-pipeline avatar May 20 '24 11:05 deep-pipeline

I think would be better close this PR because it is very old and doesn't make sense.

patrickreinan avatar Jun 07 '24 03:06 patrickreinan