Grabbing multiple subsets for the same table?

[reidab]

I've got a use case where I'm interested in grabbing:

1. Unmodified copies of the user records for our staff
2. Anonymized copies of some non-staff users

I tried defining two [[Tables]] blocks in the config, but only the first is executed.

[[Tables]]
  Name = "users"
  IgnoreData = false

  [Tables.Filter]
    Match = "AdminUsers"

[[Tables]]
  Name = "users"
  IgnoreData = false

  [Tables.Filter]
    Match = "Latest100Users"

  [Tables.Anonymise]
    email = "EmailAddress"
    first_name = "FirstName"

I also tried using two separate config files, but I can't see a way to process only data and not the structure, so I run into an error because the tables already exist.

Is there a way to do this currently?

[Aj-vrod]

Hi @reidab Thanks for reporting this. Unfortunately, at the moment this is not possible to do. I will take a look and see how to proceed to improve it :)

[renatosuero]

Hey folks, I'm testing the tool and I have the same situation. I was wondering how it could be fixed. We have IgnoreData to only create the structure. What about IgnoreStructure to only insert the data. When I try to anonymise the same table two times I have the error the table already exists. Using the flag to ignore the structure it could allow us to have mutiple rules in the same config file

[[Tables]]
  Name = "users"
  [Tables.Filter]
    Match = "AdminUsers"

[[Tables]]
  Name = "users"
  IgnoreStructure = true

  [Tables.Filter]
    Match = "Latest100Users"

  [Tables.Anonymise]
    email = "EmailAddress"
    first_name = "FirstName"

I've seen we already have the option --data-only, but in that case we need to have multiples steal process. The option is amazing, but it is "global" config, using the option into the table config it could be more handy for that cases.

I can work on that if you thing it could be a good addition to the tool

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had any activity in the last 21 days. Feel free to re-open in case you would like to follow up.