torcharrow
torcharrow copied to clipboard
pytorch
High performance model preprocessing library on PyTorch
For single colulmn, delegating to Arrow Array seems to be a good initial support. Similar to https://github.com/facebookresearch/torcharrow/issues/64 and https://github.com/facebookresearch/torcharrow/issues/53 Arrow arrays supports `unique`: https://arrow.apache.org/docs/python/generated/pyarrow.compute.unique.html#pyarrow.compute.unique ```python >>> import pyarrow as pa...
interop.from_pylist has the dtype argument default to None, which must be a DType instance before passing to Scope.default._FromPyList. https://github.com/facebookresearch/torcharrow/blob/main/torcharrow/interop.py#L40 This issue is to fix it so a dtype is automatically...
```python >>> import torcharrow as ta >>> a = ta.Column(['a', 'b']) >>> help(a) ``` It shows some numeric related methods , although it's a string column: ``` ... | __neg__(self)...
```python col = ta.Column([{"a": 1}, {"b1": 2, "b2": 3}]) col 0 {'a': 1} 1 {'b1': 2, 'b2': 3} dtype: Map(string, int64), length: 2, null_count: 0 ``` Result for `to_arrow()`: ```...
To reproduce: ```python import torcharrow as ta df = ta.DataFrame({"a": [1, 2, 3]}) df["b"] ``` Output ``` >>> df["b"] E20211112 13:52:55.454295 207011328 Exceptions.h:68] /Users/wxie/Code/torcharrow/csrc/velox/velox/velox/type/Type.cpp273getChildIdxindex.has_value()Field not found: bUSERINVALID_ARGUMENT Traceback (most recent...
```python import torcharrow as ta import torcharrow.dtypes as dt a = ta.Column( [ [(1, 1.1)], [(2, 2.2), (3, 3.3)], [], [(4, 4.4), (5, None)], [(6, 6.6)], ], dtype=dt.List( dt.Struct( [...
Can the transformations expressed in torcharrow be compiled into Torchscript, or are there plans to support this in the future? Having a Pandas-like API that natively integrates with PyTorch seems...
`/README.md` tells us to install non-brew dependencies with the following command ``` scripts/build_mac_dep.sh ranges_v3 googletest fmt double_conversion folly re2 ``` However, it seems easier to remove the command-line options. ```...
When it comes to the type system of numerical binary operations (including the return type of the operation and the type promotion rule), there are still some discrepancy between TA's...
Consistent with other PyTorch domain libraries, e.g. https://github.com/pytorch/audio/tree/main/test and https://github.com/pytorch/data/tree/main/test