David Wood

icon indicating an email [email protected]

Results 57 comments of David Wood

add license check transform

Also, not sure how far behind dev this is. It's probably a good idea to merge with dev

[Bug] Test framework failing with "Schema of the two tables is not the same" while columns match

Thanks for the submission. However, I don't understand your reproduction script. It does not seem to use any of the data-prep-kit library. Can you provide a set of CLI or...

[Feature] pyarrow parquet write_table can save up to 30% storage with compression flag ‘ZSTD’

I believe this is fixed in #404 and #441

[Feature] pyarrow parquet write_table can save up to 30% storage with compression flag ‘ZSTD’

This is available after 0.2.0 in TransformUtils.convert_arrow_to_binary()

[Feature] Enable transform() to terminate all processing of documents across all instances

This is fixed after 0.2.0 with new UnrecoverabledException.

initial pass at a pipelining transform

```python from typing import Any import pyarrow as pa from pyarrow import Table from data_processing.transform import AbstractTableTransform from data_processing.transform.binary_pipeline import PipelinedBinaryTransform from data_processing.utils import TransformUtils class HelloTransform(AbstractTableTransform): """" Adds a...

initial pass at a pipelining transform

> This is breaking cli I have not completed that part yet. This is still a Draft afterall :)

[Feature] Enable composable/pipelined transforms in the python API

Start of formal design discussion, given recent interest in this requirement. The basic requirement is as follows then: 1. Ability to have a series of transforms called in sequence on...

[Feature] Functionality to add Hate Abuse Profanity module for NLP Data

Is this a duplicate of #383

[Feature] Filter toxic words/sentences from a given text

Is this a duplicate of #389