Hima Patel

Repo level concatenation of data

38

Can you share more details on the technique for repo level concatenation part?

Can you share some details on what languages do you do syntax checking and what tools do you use?

Dedup of code during data prep

6

Thank you for sharing the details on this work. This is indeed impressive! It was mentioned that a repo level dedup was performed. Did you guys consider exact and fuzzy...

Data prep for LLM application builders

1

Hello! This is really cool collection. I want to propose adding https://github.com/IBM/data-prep-kit as tools for quick data preparation as it may help LLM app developers started quickly. Our team developed...

### Search before asking - [X] I searched the [issues](https://github.com/IBM/data-prep-lab/issues) and found no similar issues. ### Component Transforms/Other ### Feature Identify language of each text with confidence score. ### Are...

### Search before asking - [X] I searched the [issues](https://github.com/IBM/data-prep-lab/issues) and found no similar issues. ### Component Transforms/Other ### Feature Add capability to derive quality of NLP documents. ### Are...

### Search before asking - [X] I searched the [issues](https://github.com/IBM/data-prep-lab/issues) and found no similar issues. ### Component Other ### Feature Two requests: 1) Python and ray modules need to be...

[Feature] Create new transform to ingest markdown (.md) files and convert to parquet format.

2

### Search before asking - [X] I searched the [issues](https://github.com/IBM/data-prep-lab/issues) and found no similar issues. ### Component Transforms/Other ### Feature Convert .md files to parquet files so that they can...

[Feature] Enhance code2parquet to support instruction tuning pairs as an input for data prep

2

### Search before asking - [X] I searched the [issues](https://github.com/IBM/data-prep-lab/issues) and found no similar issues. ### Component Tools/ingest2parquet ### Feature Ability to read instruction pairs with the assumption that they...

### Search before asking - [X] I searched the [issues](https://github.com/IBM/data-prep-lab/issues) and found no similar issues. ### Component Tools/ingest2parquet ### Feature Enhancement in readme to help the user to add their...