Feature Request: expandNestedObjects functionality with keys

[tatethurston]

Hey 👋 Thanks for the great library.

It would be nice to have a configuration option similar to expandNestedObjects when using keys.

I have a data with various nested fields, eg:

{ foo: 'foo', bar: 'bar', baz: { a: 'a', b: 'b' }

Where I would like to format the column naming and order using keys. I would additionally like to render a subset of fields, but expand the keys for any nested objects.

json2csv(rows, { keys }) gets close to this, but keys with a nested object value are serialized a stringified json in the column. Having an opt-in for field auto-detection for nested objects when using keys would be great, or supporting something like field: some-field-name.*.

I can work around this with something like this:

  const keys = columns.flatMap((col) => {
    const column = {
      field: col.metadata.fieldName,
      title: col.label,
    };

    const objectColumnData = rows.find((r) => {
      const val = (r as any)[column.field];
      return val && typeof val === 'object' && !Array.isArray(val);
    });

    if (objectColumnData) {
      const nestedFieldsWithoutTypename = Object.keys(
        (objectColumnData as any)[column.field],
      )
        .filter((key) => key !== '__typename')
        .map((k) => ({
          field: `${column.field}.${k}`,
          title: `${column.title}${k[0].toUpperCase() + k.slice(1)}`,
        }));
      return nestedFieldsWithoutTypename;
    }
    return [column];
  });

But a configuration option would be ideal.

[mrodrig]

Hi @tatethurston, thanks for reporting this. I can definitely see the use case for this and agree this would be a nice feature. It seems like this should be relatively straightforward to support by seeing if the provided keys match any of the auto-detected keys and then those could be included.

I have some of the code changes for this setup locally based on my initial thought of just enabling that as the default behavior, but I could also see how that might cause unintended keys to be included in certain cases, so I like your idea of having this behind an option flag. The code changes I have locally still need some work though since a couple of the existing tests are failing after I moved some logic around. I'll keep working on getting this setup and testing it to make sure it behaves as intended in different cases and will let you know as soon as I have any updates.

[tatethurston]

Amazing 🙌 , thanks @mrodrig

[mrodrig]

Hi @tatethurston, thanks again for reporting this. I just published version 5.3.0 which adds a new option (wildcardMatch) to the keys option's Object type { field: string, title?: string, wildcardMatch?: boolean }. I went with this approach since it's possible an Object key could legitimately contain *, though it certainly seems like more of an edge case, so having a separate way to specify which keys should be expanded helps ensure that's the desired behavior.

Here's an example of it in action:

const { json2csv } = require('json-2-csv');

const list = { foo: 'foo', bar: 'bar', baz: { a: 'a', b: 'b' } };

const keys = ['foo', 'bar', { field: 'baz', wildcardMatch: true }];

const csv = json2csv(list, { keys })
console.log(csv);

and the resulting CSV:

foo,bar,baz.a,baz.b
foo,bar,a,b

Hopefully this helps!

[tatethurston]

Thanks @mrodrig!. Two things I noticed that would improve this option from my POV:

• excludeKeys doesn't apply to these nested fields, but it would be great if it did similar to arrays.
• it would be awesome if the title used for key was preserved for the nested keys. Eg:

{ title: "Foo", field: 'foo', wildcardMatch: true } would be something like Foo.subfield instead of foo.subfield

[mrodrig]

Happy to help @tatethurston! Thanks for mentioning that. From my testing, it seems that excludeKeys is overriding the keys option like I'd expect, but maybe there's an edge case I'm not hitting/thinking of. Could you provide an example where you're seeing that behavior?

I like the idea of using the titles as a prefix for the matched nested keys too. I hadn't considered that, but I think that would be entirely possible to support. I'll add it to the list of things I'm planning to take a look at - thanks!

[tatethurston]

Happy to help @tatethurston! Thanks for mentioning that. From my testing, it seems that excludeKeys is overriding the keys option like I'd expect, but maybe there's an edge case I'm not hitting/thinking of. Could you provide an example where you're seeing that behavior?

I like the idea of using the titles as a prefix for the matched nested keys too. I hadn't considered that, but I think that would be entirely possible to support. I'll add it to the list of things I'm planning to take a look at - thanks!

Given { foo: { __typename: 'something', bar: 'baz' } }, when using { keys: [{ field: 'foo', title: 'Foo', wildcardMatch: true}], excludeKeys: '__typename' } I expected to see foo.bar\nbaz but instead saw foo.__typename,foo.bar\nsomething,baz. Possible I'm overlooking something.

The output that I'm ultimately building towards given the above input is Foo Bar\nbaz.

[mrodrig]

Oh, it looks like the reason it isn't excluding the key is because excludeKeys takes the key paths to be excluded. In your example JSON, that would be:

excludeKeys: ['foo.__typename']

That should help get a bit closer to the desired output. The field title mapping is something that I haven't been able to get to, but will see if I can possibly get to it in the coming weeks.

[tatthurs]

Oh, it looks like the reason it isn't excluding the key is because excludeKeys takes the key paths to be excluded. In your example JSON, that would be:

excludeKeys: ['foo.__typename']

That should help get a bit closer to the desired output. The field title mapping is something that I haven't been able to get to, but will see if I can possibly get to it in the coming weeks.

Ah, thank you. I want to avoid this field across all objects (and nested objects) so it looks like I would do something like this:

excludeKeys: ['__typename', ...keys.map(key => `${key.field}.__typename`)]

[mrodrig]

Happy to help. 🙂 You bring up a great point with that example use case though. When I setup excludeKeys matching, I only anticipated use cases where a prefix would be specified to exclude all keys under a certain path, but wanting to exclude all keys with a certain name regardless of where they appear is a completely valid use case too. I'm thinking that it might make sense at this point to update excludeKeys to support either an array of strings or RegExps to help handle this use case. I'll have to take a closer look to see how I can support that in a backwards compatible manner.

[tatthurs]

I'm thinking that it might make sense at this point to update excludeKeys to support either an array of strings or RegExps to help handle this use case

Yeah that would be pretty slick. The first API that came to mind was /*.__typename/ for this case. It looks like prefix matching is already supported:

excludeKeys - Array - Specify the keys that should be excluded from the output. Provided keys will also be used as a RegExp to help exclude keys under a specified prefix, such as all keys of Objects in an Array when expandArrayObjects is true.

It's just these nested keys were it gets more interesting.