fides icon indicating copy to clipboard operation
fides copied to clipboard

BigQuery nested fields are generated flat

Open SteveDMurphy opened this issue 2 years ago • 0 comments

Bug Description

When generating a dataset from BigQuery, the output dataset is currently flat instead of being appropriately nested

Steps to Reproduce

  1. Generate the test dataset from BigQuery (creds in 1Password)
  2. Inspect the dataset and note that the dot-separated columns are treated as top level columns although they should be treated as a nested structure

Expected behavior

The generated dataset should nest fields appropriately under their parent

Screenshots

image

Environment

  • Version: 1.7.0
  • OS: Darwin
  • Python Version: 3.9.12
  • Docker Version: 20.10.16

Additional context

This was discovered as part of the classify work and noted as a potential issue for fidesctl. Tested today to confirm the issue

SteveDMurphy avatar Jul 28 '22 02:07 SteveDMurphy