improve: generalize transformations and scripts of runner and preloads into TemplateTransformer

[bowenliang123]

Description

• generalize scripts of runner and preload into TemplateTransformer
• properly indent scripts with dedent inside methods

Type of Change

Please delete options that are not relevant.

• [ ] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] This change requires a documentation update, included: Dify Document
• [x] Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
• [ ] Dependency upgrade

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• [x] existed tests for code executor

Suggested Checklist:

• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] My changes generate no new warnings
• [x] I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
• [ ] optional I have made corresponding changes to the documentation
• [ ] optional I have added tests that prove my fix is effective or that my feature works
• [ ] optional New and existing unit tests pass locally with my changes

[bowenliang123]

cc @takatost @Yeuoly

[bowenliang123]

BTW, during the refactoring, it's found that the approaches for the input objects serialization vary between Python and Javascript. The one for Python is wrapped in Base64 encoding, while the Javascript uses the raw inputs json object. Not the best way for further extensions to other languages. Is that as designed on purpose?

[Yeuoly]

BTW, during the refactoring, it's found that the approaches for the input objects serialization vary between Python and Javascript. The one for Python is wrapped in Base64 encoding, while the Javascript uses the raw inputs json object. Not the best way for further extensions to other languages. Is that as designed on purpose?

In Python, the boolean value True is capitalized, which is not standard in JSON encoding. As a result, simple JSON encoding might not work as unexpected. However, this issue does not occur in JavaScript.

[bowenliang123]

Yes, the differences were noticed. So maybe we could unify the json+base64 as the standard serialization for input objects in the future.

[bowenliang123]

And there's another point I may want to have your suggestions. We are still using replace with to fill the code and inputs, as the existed way before refactoring. Do you want jinja2 to do this for possible injection attacks (but I'm not sure how much MarkupSafe inside jinja2 does help) ?

[Yeuoly]

Yes, the differences were noticed. So maybe we could unify the json+base64 as the standard serialization for input objects in the future.

feel free to do it~

[Yeuoly]

And there's another point I may want to have your suggestions. We are still using replace with to fill the code and inputs, as the existed way before refactoring. Do you want jinja2 to do this for possible injection attacks (but I'm not sure how much MarkupSafe inside jinja2 does help) ?

It's unnecessary I think, the replace method doesn't significantly impact performance, maybe format could be a good alternative. As for Jinja2, it's too heavy for this task. It supports many features like loops, which we don't need. Additionally, it introduces some security risks. Even if it's currently safe, security experts can always find new ways to exploit it as I faced before.

[bowenliang123]

And there's another point I may want to have your suggestions. We are still using replace with to fill the code and inputs, as the existed way before refactoring. Do you want jinja2 to do this for possible injection attacks (but I'm not sure how much MarkupSafe inside jinja2 does help) ?

It's unnecessary I think, the replace method doesn't significantly impact performance, maybe format could be a good alternative. As for Jinja2, it's too heavy for this task. It supports many features like loops, which we don't need. Additionally, it introduces some security risks. Even if it's currently safe, security experts can always find new ways to exploit it as I faced before.

Yes, I agree with you. Let's keep it with "replace".

[bowenliang123]

Yes, the differences were noticed. So maybe we could unify the json+base64 as the standard serialization for input objects in the future.

feel free to do it~

Hi, I have get this done. json + base64 serialization is now a shared mechanism for both Python and Javascript. No blocker issues or discussions remain in this PR.

[loganclark360]

👍

W. Logan Clark

On Sun, May 19, 2024 at 1:49 AM Bowen Liang @.***> wrote:

Yes, the differences were noticed. So maybe we could unify the json+base64 as the standard serialization for input objects in the future.

feel free to do it~

Hi, I have get this done. json + base64 serialization is now a shared mechanism for both Python and Javascript.

— Reply to this email directly, view it on GitHub https://github.com/langgenius/dify/pull/4487#issuecomment-2119156394, or unsubscribe https://github.com/notifications/unsubscribe-auth/BH6TQ24YBKSFBUTF2TPZOOTZDBRS7AVCNFSM6AAAAABH4CLMXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJZGE2TMMZZGQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>