CSV-based bulk update functionality

[bhunt64848]

NetBox version

v2.10.4

Feature type

Change to existing functionality

Proposed functionality

Being able to import serial numbers or any given field without having the export delete and re-import would be absolute amazing (wanted to add 45 serial numbers and couldn’t do it)

Use case

Provide efficiency in mass updating the database

Database changes

No response

External dependencies

No response

[DanSheps]

I think a better way to phrase this would be "bulk update via import"

[bhunt64848]

Agreed!

[Mizarv]

This has already been suggsted in #1732 with a reply from Jeremy.

Given it was 4 years ago, I would love to see this looked at again.

[jeremystretch]

I'm afraid this needs quite a bit more detail. What is the proposed workflow? How will you accurately identify existing objects?

[Mizarv]

I guess now that we can export the CSVs with the object ID you could use that as that field can't be updated.

Maybe an additional tab to avoid complicating the existing CSV Data method as this works excellently

[martinum4]

Maybe it could the same as the normal csv import in regard to attribute extension and selection, example:

device.id,serial,asset_tag 
500,0815,47111234

would be used to update the serial and asset tags for the device with ID 500. However it could also be like

device,serial,asset_tag 
TestDevice,0815,47111234

since the name is an unique identifier too.

The next page could be similar to the interface-renaming screen where the prechange and postchange-data for each device is displayed, that way it could be avoided to update already set fields.

[jeremystretch]

Implementing this would require an absolutely reliable unique object identifier, i.e. the object's primary key (numeric ID). If that's acceptable, I think this is reasonably feasible to implement. However, it would likely require a separate view and workflow from the bulk import function, as we would be updating objects rather than creating them, and none of the form fields would be required (similar to the existing bulk edit functionality).

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our contributing guide.

[mikewilliamsjr]

Would addressing this issue include ensuring that the data that is exported can be immediately imported once the data is deleted, or can be imported to a fresh instance without curation? Currently, bulk import demands headers formatted in lower-case and requires that some content is in a specific format. If data must be imported in a specific format, the data should be exported in the same format as what is acceptable for import. Usability will be improved and admin effort will be reduced.

[jeremystretch]

Would addressing this issue include ensuring that the data that is exported can be immediately imported once the data is deleted, or can be imported to a fresh instance without curation?

No. This has been discussed many times and in many cases is not feasible due to import requiring sufficiently unique identifiers for related objects, which would be extraneous in exported data.

[mikewilliamsjr]

Would addressing this issue include ensuring that the data that is exported can be immediately imported once the data is deleted, or can be imported to a fresh instance without curation?

No. This has been discussed many times and in many cases is not feasible due to import requiring sufficiently unique identifiers for related objects, which would be extraneous in exported data.

You've acknowledged that including the identifiers in imported data would make martinum4's suggestion feasible. How is their suggestion materially different from mine, aside from their pure focus on importing updates? Support for updating existing records is effectively the same as importing new, unique records. The same process for ensuring primary keys are not duplicated could be used in both instances.

The logical next step is to provide a means for easily accessing the primary keys. A toggled setting to include/exclude the keys on export could improve the usability of this tool, especially if your intent is to develop a new workflow for bulk update. The keys are only extraneous if the intent is to use exported data in some other application with no intent to import; if the intent is to migrate to a fresh instance or programmatically update existing data, then rather than being extraneous, the keys seem mandatory.

[jeremystretch]

How is their suggestion materially different from mine, aside from their pure focus on importing updates?

A primary key don't exist until the object has been created in the database. It is not possible to reference one until after the object has been imported.

Let's please keep this discussion limited to the scope of the FR, which involves the bulk update of existing objects.

[mikewilliamsjr]

A primary key don't exist until the object has been created in the database. It is not possible to reference one until after the object has been imported.

RIght; hence, the suggestion to add an option to include the key in exported data.

The point I'm making is that both requests can be supported with a more useful export. IOW, by making it possible to include the keys for each record with the exported data, you would enable users to export the data, manipulate the data as required, then either import to the existing instance (update) or to a new instance.

Let's please keep this discussion limited to the scope of the FR, which involves the bulk update of existing objects.

Please forgive me for not realizing how I'm offtopic. Unless my understanding is far off, it seems like the path to CSV-based bulk updates would include a means for retrieving the primary key of each record. From there, the next logical step seems to be to ensure that exported column headers match the required column headers for import. Each of these steps should simplify the logic and/or administrative effort required for data validation.

[jdavidson2021]

Maybe have two choices during the CSV export to provide the data with "friendly names" from what is currently shown in the GUI (current design) and 2nd choice to export the data with "import column names". We currently export the data in CSV, delete the objects, massage all of the column names that won't import correctly from the current CSV export, update our data in the CSV, and then re-import the data. I'm sure this is what most users are doing when they need to do a bulk update of information currently.

[jeremystretch]

To reiterate, this FR has nothing to do with object export; the functionality required to export object IDs is already present. The work necessary to implement this FR involves:

• Extending the CSV import forms to accept a primary key identifying an existing object
• Modifying the CSV form to relax field validation (i.e. required fields) when updating existing objects
• Extending the import logic to instantiate an existing object from the database when a PK is specified (or else default to instantiating a new object)
• Modifying the view logic to create or update objects as necessary

(This is assuming the above is feasible. If not, we'll need to introduce a new form & view to accommodate the functionality.)

[jnovak-netsystemcz]

Hi,

I'm focused on update of existing items. I tested it and it works, but I have proposal:

Import description in 'Field Options' states that 'id' is required for update, but I'm afraid it is not emphasized. I propose to write it to the top of description. I mean add above 'Field Options' something like:

"If you would like update existing items, you must add 'id' column." and left 'id' in list of fields.

To be honest, I propose to split import and update as separate actions. Name import as 'import new' and there will be no 'id' in the list. For update, there will 'id' as first item in the list and will be marked as mandatory. In that case no sentence proposed above is needed.

I know that users can complain that they might need import and update at same time, but I think it is easy to split input file to two parts - new items and updated items and run it in two steps. For newcomers it will be much straightforward.