data-prep-kit icon indicating copy to clipboard operation
data-prep-kit copied to clipboard

Published 20 hours ago • verified publisher icon IBM

[Feature] Enhance code2parquet to support instruction tuning pairs as an input for data prep

Open Bytes-Explorer opened this issue 1 year ago • 2 comments

Search before asking

  • [X] I searched the issues and found no similar issues.

Component

Tools/ingest2parquet

Feature

Ability to read instruction pairs with the assumption that they are in JSON format.

Are you willing to submit a PR?

  • [X] Yes I am willing to submit a PR!

Bytes-Explorer avatar May 23 '24 05:05 Bytes-Explorer

We need to enhance the transform such that every instruction pair becomes one row in the parquet files.

Bytes-Explorer avatar Jun 27 '24 07:06 Bytes-Explorer

this would now be appled to code2parquet which superseded ingest2parquet which is not deprecated.

daw3rd avatar Sep 13 '24 16:09 daw3rd