si icon indicating copy to clipboard operation
si copied to clipboard

feat(luminork): add bulk component operations endpoints

Open jhelwig opened this issue 1 month ago • 1 comments

Problem

The Luminork API currently requires separate HTTP requests for each component operation (create, update, delete, erase). When users need to perform operations on multiple components—such as provisioning infrastructure with dozens of resources, bulk updating configuration, or cleaning up test environments—this creates significant overhead:

  • Network inefficiency: While clients can parallelize requests, they must manage concurrency limits and batch serial groups of parallel requests, adding complexity
  • Transaction coordination: Multiple requests cannot be atomic without complex client-side transaction management
  • Error handling complexity: Partial failures require clients to track which operations succeeded and implement rollback logic
  • API request overhead: Each request carries full HTTP overhead (headers, auth, connection setup)

This becomes particularly problematic for:

  • Infrastructure-as-code scenarios provisioning complete environments (10-100+ components)
  • Template instantiation creating interconnected component graphs
  • Test cleanup requiring deletion of many components
  • Bulk configuration updates across component sets

Impact

Users working with multiple components face:

  • Complex parallelization: Must implement batching logic to respect concurrency limits while maximizing throughput
  • Difficult error recovery: Partial failures in parallel batches require complex client-side rollback logic
  • No transaction atomicity: Related operations across multiple requests cannot guarantee all-or-nothing semantics
  • Increased API load: High request volume even with parallelization

This particularly affects:

  • Automated tooling (CI/CD, infrastructure provisioning scripts)
  • Large-scale testing scenarios
  • Users working with templates that instantiate many components
  • Operations on component hierarchies with managed relationships

Solution

This PR adds four new bulk operation endpoints to the Luminork V1 API:

  1. `POST /components/create_many`: Create multiple components in a single transaction
  2. `PUT /components/update_many`: Update multiple components atomically
  3. `DELETE /components/delete_many`: Soft-delete multiple components (marks for deletion, queues actions)
  4. `POST /components/erase_many`: Hard-delete multiple components (immediate removal, no actions)

Implementation approach:

  • Extracted core operations (`operations.rs`): Shared business logic for component operations used by both single and bulk handlers
  • Refactored existing handlers: `create_component` and `update_component` now use the shared core logic
  • Fail-fast error handling: Bulk operations stop on first error, returning the index of the failing operation
  • Transactional semantics: All operations within a request succeed or fail atomically (no partial application)
  • Performance optimizations: Shared data structures (component lists for managed_by resolution, head component sets) fetched once and reused

After this change, users can perform bulk operations with:

  • Single HTTP request for N operations
  • Atomic transaction guarantees
  • Clear error reporting (failed operation index + underlying error)
  • Identical semantics to single-operation equivalents

Design Rationale

Why fail-fast vs. partial success?

  • Matches SDF API semantics: Bulk endpoints in SDF API use fail-fast behavior; Luminork maintains consistency
  • Data consistency: Partial failures leave system in undefined state
  • Simpler error handling: Clients know operation either fully succeeded or fully failed
  • Transaction semantics: Matches database transaction expectations (all-or-nothing)
  • Clear debugging: Index-based error reporting pinpoints exact failure point

Why lazy-fetch component list?

  • Component list only needed when `managed_by` specified
  • Bulk create_many with no managed relationships avoids expensive list operation
  • Fetch once if needed, reuse across all operations in batch
  • Reduces overhead for common case (no management relationships)

Why same route prefix (`/components/*_many`)?

  • Natural API discoverability (bulk variants adjacent to single operations)
  • Consistent with existing Luminork API patterns
  • Clear semantic relationship (create → create_many, update → update_many)
  • Each bulk endpoint mirrors the semantics of its single-operation counterpart

Deployment Notes

API changes:

  • Four new endpoints under `/v1/w/{workspace_id}/change-sets/{change_set_id}/components/`
  • No changes to existing endpoints (backward compatible)
  • All bulk operations require non-HEAD change set (returns 400 BAD_REQUEST on HEAD)

Monitoring:

  • PostHog events: `api_create_many_components`, `api_update_many_components`, `api_delete_many_components`, `api_erase_many_components`
  • Each event includes `count` field with number of operations in batch
  • Existing single-operation audit logs maintained (queued per-component, published on commit)

Performance characteristics:

  • Single transaction for entire batch (all-or-nothing commit)
  • Shared data structures reused across operations (component lists, head component sets)
  • Linear processing (sequential operations within batch)
  • Network overhead reduced from O(N requests) to O(1 request)

Compatibility

  • No breaking changes: Existing single-operation endpoints unchanged
  • API additions only: New bulk endpoints are additive
  • Semantic consistency: Bulk operations behave identically to N individual operations within a transaction

jhelwig avatar Nov 17 '25 19:11 jhelwig

Dependency Review

✅ No vulnerabilities or OpenSSF Scorecard issues found.

Scanned Files

None

github-actions[bot] avatar Nov 17 '25 19:11 github-actions[bot]