ckan icon indicating copy to clipboard operation
ckan copied to clipboard

update resources faster with package_update

Open wardi opened this issue 1 year ago • 0 comments

Fixes #5713

Proposed fixes:

  1. selectively skip validation and dictization on resources if both are true:
    • dataset metadata is unchanged
    • skipped resource metadata is unchanged
  2. skip updates completely if no metadata was changed
  3. compare new and existing resource model objects before commit to avoid commit processing time
  4. pass original_package in context from actions that call package_show with for_update=True to package_update to allow low-cost validated metadata comparisons
100-resource dataset action old new improvement
resource_create .721 .419 70%
resource_update 1.05 .445 130%
resource_delete .709 .478 48%
package_resource_reorder .697 .479 45%
package_patch .660 .621 6% *
resource_patch .746 .413 80%
package_revise (updating resource) .787 .267 190%

* validation and dictization applied for all resources when any dataset field changes, we only save commit time

[!NOTE] This change will affect custom validation rules that access resource metadata from dataset or other resource validators:

  • only the changed resources are passed to validation
  • flattened data the validators receive won't include skipped resources

This change does not affect validation of resource fields that depend on package fields because the package fields are always available in flattened data

Features:

  • [x] includes tests covering changes
  • [ ] includes updated documentation
  • [ ] includes user-visible changes
  • [ ] includes API changes
  • [ ] includes bugfix for possible backport

wardi avatar Jul 22 '24 14:07 wardi