ckan
ckan copied to clipboard
update resources faster with package_update
Fixes #5713
Proposed fixes:
- selectively skip validation and dictization on resources if both are true:
- dataset metadata is unchanged
- skipped resource metadata is unchanged
- skip updates completely if no metadata was changed
- compare new and existing resource model objects before commit to avoid commit processing time
- pass
original_packagein context from actions that callpackage_showwithfor_update=Truetopackage_updateto allow low-cost validated metadata comparisons
| 100-resource dataset action | old | new | improvement |
|---|---|---|---|
| resource_create | .721 | .419 | 70% |
| resource_update | 1.05 | .445 | 130% |
| resource_delete | .709 | .478 | 48% |
| package_resource_reorder | .697 | .479 | 45% |
| package_patch | .660 | .621 | 6% * |
| resource_patch | .746 | .413 | 80% |
| package_revise (updating resource) | .787 | .267 | 190% |
* validation and dictization applied for all resources when any dataset field changes, we only save commit time
[!NOTE] This change will affect custom validation rules that access resource metadata from dataset or other resource validators:
- only the changed resources are passed to validation
- flattened data the validators receive won't include skipped resources
This change does not affect validation of resource fields that depend on package fields because the package fields are always available in flattened data
Features:
- [x] includes tests covering changes
- [ ] includes updated documentation
- [ ] includes user-visible changes
- [ ] includes API changes
- [ ] includes bugfix for possible backport