[BUG] NVTabular Dataset constructor cannot process cudf.StructType values.

[drobison00]

Describe the bug Attempting to create an NVT Dataset using a cudf DataFrame containing a struct dtype fails.

Steps/Code to reproduce bug

Create a test file:

echo '[{"properties":{"id":"bddc03c8-8da3-4ef6-8ef9-a50324639100", "location":{"city":"Port Denisetown","state":"Smithton","countryOrRegion":"XR","geoCoordinates":{"latitude":3.5518965,"longitude":131.871582}}}}]' > example.json

reproducer.py

import cudf
import nvtabular as nvt

df = cudf.read_json("example.json")
print(f"DataFrame dtypes: \n{df.dtypes}")

ds = nvt.Dataset(df)

output

properties    struct
dtype: object

Traceback (most recent call last):
... SNIP ...
TypeError: Merlin doesn't provide a mapping from struct (<class 'cudf.core.dtypes.StructDtype'>) to a Merlin dtype. If you'd like to provide one, you can use `merlin.dtype.register()`.

Expected behavior Since its a standard cuDF data type, I'd expect it to be processed correctly by NVT, or some type of graceful fallback behavior.

Environment details (please complete the following information):

• Conda environment

conda list | grep 'nvtabular|merlin'
merlin-core               23.02.01                   py_0    nvidia
merlin-dataloader         23.02.01                   py_0    nvidia
nvtabular                 23.02.00                 py38_0    nvidia

Additional context Add any other context about the problem here.

[drobison00]

Updated repro that illustrates workflow issues in addition to Dataset creation.

def f_to_pandas(col, df):
    pd_series = col.to_pandas()

    return cudf.from_pandas(pd_series)

def test_cudf_struct_type_conversion():
    import cudf
    import nvtabular as nvt
    from nvtabular.ops import LambdaOp
    from nvtabular.ops.operator import ColumnSelector

    input_df = cudf.read_json("example.json")  #  different error if we use pd.read_json

    single_op = ColumnSelector("properties") >> LambdaOp(f=f_to_pandas)
    workflow = nvt.Workflow(single_op)

    ds = nvt.Dataset(input_df)
    result = workflow.fit_transform(ds).to_ddf().compute()

    print(result)

[karlhigley]

This is related to a lower-level issue that happens when converting cuDF struct columns that contain both nulls and empty structs to Pandas. It can be worked around by exploding structs into separate columns with series.struct.explode() before passing data into NVT.

[drobison00]

This issue should be fully resolved when https://github.com/rapidsai/cudf/pull/13315 goes in.

[rnyak]

@drobison00 hello! is the issue solved at your end. looks like https://github.com/rapidsai/cudf/pull/13315 was merged.

[drobison00]

@rnyak I'll double check today.