dust icon indicating copy to clipboard operation
dust copied to clipboard
verified publisher icon dust-tt

Support importing libraries (or functions) in a CODE block

Open okomarov opened this issue 2 years ago • 3 comments

Adding support for external dependencies in code blocks would be great for pre/post-processing steps.

A couple of example would be:

  • https://www.npmjs.com/package/html-to-text to convert html input into a more readable format while preserving links
  • https://www.npmjs.com/package/json-difference to compare expected versus obtained results

okomarov avatar Apr 23 '23 13:04 okomarov

Yea that would be great. We’ll slowly converge to that by removing js_sandbox and integrating directly to Deno 👍

spolu avatar Apr 23 '23 14:04 spolu

Hey, I'm currently using Dust.tt to test workflows before implementing them into LangChain or my own code. I believe that with ChatGPT or GPT-4, it's pretty straightforward to create your own preprocessing and postprocessing functions in other languages. I preprocess the text into JSONL format, and then process the results returned by the API according to my preferences. In this case, I transform the data into a format suitable for my database. If you need help with scripting for preprocessing or postprocessing, feel free to leave a comment and I'll assist you.

cmirdesouza avatar Jun 08 '23 14:06 cmirdesouza

@cmirdesouza Ideally I can prototype the full pipeline and get a comprehensive report of the regressions in dustt than having multiple separate steps.

okomarov avatar Jun 08 '23 17:06 okomarov