[Feature] Implement a super-pipeline for Code processing

[roytman]

trafficstars

Search before asking

• [X] I searched the issues and found no similar issues.

Component

KFP workflows

Feature

Implement a super pipeline that will combine all code-processing steps.

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

[roytman]

@himapatel1 , please provide the Code processing order and maybe the input data.

[Bytes-Explorer]

Lets follow what we have here https://github.com/IBM/data-prep-kit/tree/dev/examples for order. All the existing test datasets we have can be used. Let me know if you need me to point you to them.

[revit13]

Lets follow what we have here https://github.com/IBM/data-prep-kit/tree/dev/examples for order. All the existing test datasets we have can be used. Let me know if you need me to point you to them.

@Bytes-Explorer I do not see malware there... where it should be placed? Thanks

[Bytes-Explorer]

@revit13 It can be placed right after code quality.

[Bytes-Explorer]

@revit13 @roytman It would be nice if a user can change the order as per their wish. They can provide input in a JSON. Is that kind of automation possible? If this is more work, we can create a new issue for it and you can also leverage Arun/Sai's help.

[roytman]

@revit13 @roytman It would be nice if a user can change the order as per their wish. They can provide input in a JSON. Is that kind of automation possible? If this is more work, we can create a new issue for it and you can also leverage Arun/Sai's help.

@Bytes-Explorer , I work with Arun and Sai. The automation pipeline generation is in next steps. In the current version, user is able to skip steps that he would like to.

[Bytes-Explorer]

Thanks Alexey, lets make a new issue for the automation pipeline generation, so that we dont miss it.

[roytman]

See https://github.com/IBM/data-prep-kit/issues/182