Details

Since we introduced the WorkOrder module, the existence of Invocation doesn't fit anymore. If anything, I propose the follow models are moved under WorkOrder.

Run
Step
StepRun
Dataclip (maybe?)
...

This implies the removal of the Invocation module afterwards.

Implementation notes

I also think we should revert the spelling of WorkOrder to Workorder, the underscore variable naming is tedious and used in so many places that it's visually unnecessary.

Release notes

User acceptance criteria

Dec 12 '22 06:12 stuartc

This would be a welcome change (goal to remove Invocation), but I'd like to spend a small amount of time thinking about what this means (or what it should mean) now - the original issue is over 2 years old (I did just update the module names).

Jul 17 '25 17:07 stuartc

@stuartc , mapped this out with Claude and think it's soon going to be time to rip the bandaid off. After the React re-write is done, let's discuss, refresh, and execute this plan:

Refactor Plan: Remove Lightning.Invocation Module

Author: Taylor Downs Date: 2025-10-19 Status: Planning

Executive Summary

Since the introduction of the WorkOrder module, the Lightning.Invocation module no longer fits the domain model. This refactor will reorganize execution-related models under Lightning.WorkOrders and extract dataclip functionality to its own context, then remove Lightning.Invocation entirely.

Goals

Move execution models under Lightning.WorkOrders context
Create dedicated Lightning.Dataclips context
Remove Lightning.Invocation module completely
Maintain all existing functionality with no behavior changes
Improve code organization and domain clarity

Domain Model After Refactor

Lightning.WorkOrders (Execution Hierarchy)
├── WorkOrder (schema)
├── Run (schema) - moved from Invocation
├── Step (schema) - moved from Invocation
├── RunStep (schema) - moved from Invocation
├── LogLine (schema) - already here conceptually
└── Query (queries for all above)

Lightning.Dataclips (Independent Data Artifacts)
├── Dataclip (schema) - moved from Invocation
├── DataclipAudit (schema) - moved from Invocation
└── Query (queries for dataclips)

Lightning.Runs (removed - merged into WorkOrders)

Rationale

Why move Run, Step, RunStep under WorkOrders?

Natural Hierarchy:

WorkOrders are the top-level execution container
Each WorkOrder has one or more Runs (original attempt + retries)
Each Run has one or more Steps (job executions)
RunStep is the join table linking runs to steps

Single Responsibility:

WorkOrders context manages the entire execution lifecycle
All execution queries and operations live in one place
Clear ownership of execution state management

Why keep Dataclips separate?

Independence:

Dataclips can exist without WorkOrders (global dataclips, saved inputs)
Dataclips are referenced by multiple contexts (WorkOrders, Runs, Manual runs)
Dataclips have their own lifecycle (creation, naming, wiping, retention)

Domain Concept:

Dataclips represent data artifacts that flow through execution
They're inputs and outputs, not execution records themselves
Similar to how Credentials exist independently but are used by Jobs

File Structure Changes

Current Structure

lib/lightning/
├── invocation.ex (446 lines) - TO REMOVE
├── invocation/
│   ├── query.ex (423 lines) - TO REMOVE
│   ├── dataclip.ex - MOVE to dataclips/
│   ├── dataclip_audit.ex - MOVE to dataclips/
│   ├── step.ex - MOVE to work_orders/
│   ├── log_line.ex - MOVE to work_orders/
│   └── run_step.ex - MOVE to work_orders/
├── work_orders.ex (794 lines) - EXPAND
├── work_orders/
│   └── (existing files)
├── runs.ex (384 lines) - MERGE into work_orders.ex
├── runs/
│   └── (merge into work_orders/)
├── run.ex (schema) - MOVE to work_orders/

Target Structure

lib/lightning/
├── work_orders.ex (EXPANDED - all execution operations)
├── work_orders/
│   ├── work_order.ex (schema)
│   ├── run.ex (schema) ← moved from lib/lightning/runs/run.ex
│   ├── step.ex (schema) ← moved from invocation/step.ex
│   ├── run_step.ex (schema) ← moved from invocation/run_step.ex
│   ├── log_line.ex (schema) ← moved from invocation/log_line.ex
│   ├── query.ex (ALL execution queries)
│   ├── events.ex (execution events)
│   ├── handlers.ex (execution handlers)
│   └── (other existing files)
├── dataclips.ex (NEW - all dataclip operations)
├── dataclips/
│   ├── dataclip.ex (schema) ← moved from invocation/dataclip.ex
│   ├── audit.ex ← moved from invocation/dataclip_audit.ex
│   └── query.ex (NEW - dataclip queries)

Detailed Migration Plan

Phase 1: Create New Structure (No Breaking Changes)

1.1 Create Lightning.Dataclips Context

Create lib/lightning/dataclips.ex:

defmodule Lightning.Dataclips do
  @moduledoc """
  Context for managing Dataclips.

  Dataclips represent data artifacts that flow through Lightning workflows.
  They can be:
  - Webhook payloads (http_request)
  - Step results (step_result)
  - Saved inputs (saved_input)
  - Global data (global)
  - Kafka messages (kafka)
  """

  # Move all dataclip functions from Lightning.Invocation
  # Functions to move:
  # - list_dataclips/0, list_dataclips/1, list_dataclips_query/1
  # - list_dataclips_for_job/2, list_dataclips_for_job/3
  # - list_dataclips_for_job_with_cron_state/3
  # - get_dataclip/1, get_dataclip!/1, get_dataclip_with_body!/1
  # - get_dataclip_for_run/1
  # - get_first_dataclip_for_run_and_job/2
  # - get_next_cron_run_dataclip/2
  # - cron_triggered_job?/1
  # - create_dataclip/1, update_dataclip/2, update_dataclip_name/3
  # - delete_dataclip/1, change_dataclip/2
  # - get_dataclip_query/1, get_output_dataclip_query/1
end

Create lib/lightning/dataclips/query.ex:

defmodule Lightning.Dataclips.Query do
  @moduledoc """
  Query functions for Dataclips.
  """

  # Move query functions from Lightning.Invocation.Query:
  # - dataclip_with_body/0
  # - last_n_for_job/2
  # - select_as_input_text/1
  # - wipe_dataclips/1
end

Move schemas:

lib/lightning/invocation/dataclip.ex → lib/lightning/dataclips/dataclip.ex
- Update schema "dataclips" (stays same)
- Update all internal aliases
lib/lightning/invocation/dataclip_audit.ex → lib/lightning/dataclips/audit.ex

1.2 Move Execution Schemas Under WorkOrders

Move schemas:

lib/lightning/runs/run.ex → lib/lightning/work_orders/run.ex
- Update schema "runs" (stays same)
- Update module name to Lightning.WorkOrders.Run
- Update all internal aliases (especially Dataclip references)
lib/lightning/invocation/step.ex → lib/lightning/work_orders/step.ex
- Update schema "steps" (stays same)
- Update module name to Lightning.WorkOrders.Step
- Update all internal aliases
lib/lightning/invocation/run_step.ex → lib/lightning/work_orders/run_step.ex
- Update schema "run_steps" (stays same)
- Update module name to Lightning.WorkOrders.RunStep
lib/lightning/invocation/log_line.ex → lib/lightning/work_orders/log_line.ex
- Update schema "log_lines" (stays same)
- Update module name to Lightning.WorkOrders.LogLine

1.3 Consolidate WorkOrders Functions

Expand lib/lightning/work_orders.ex:

Merge functions from:

Current Lightning.Invocation (work order search/export functions)
Current Lightning.Runs (run lifecycle functions)
Keep existing WorkOrders functions

Functions to merge from lib/lightning/invocation.ex:

# Work Order Search & Export
- get_workorders_count_limit/0
- search_workorders/1, search_workorders/3
- search_workorders_for_retry/2
- search_workorders_for_export_query/2
- count_workorders/2
- export_workorders/3
- get_workorders_by_ids/1
- with_runs/1
- All private search helpers (filter_by_*, build_search_fields_*, etc.)

# Step Functions (execution-related)
- list_steps/0, list_steps_for_project/2, list_steps_for_project_query/1
- get_step!/1, get_step_with_job!/1
- get_first_step_for_run_and_job/2
- get_step_count_for_run/1
- change_step/2
- logs_for_step/1, assemble_logs_for_step/1
- assemble_logs_for_job_and_run/2

Functions to merge from lib/lightning/runs.ex:

# All run lifecycle operations
- enqueue/1, enqueue_many/1
- claim/2, dequeue/1
- get/2, get_for_worker/1
- get_run_options/2
- get_dataclip_body/1, get_dataclip_request/1, get_input/1
- wipe_dataclips/1
- get_credential/2
- start_run/2, complete_run/2, update_run/1, update_runs/2
- append_run_log/3
- start_step/2, complete_step/2
- mark_run_lost/1, mark_steps_lost/1
- get_project_id_for_run/1
- get_log_lines/2

Consolidate lib/lightning/work_orders/query.ex:

Merge queries from:

lib/lightning/invocation/query.ex (work order, run, step queries)
lib/lightning/runs/query.ex (run queries)

Keep organized by entity:

defmodule Lightning.WorkOrders.Query do
  # Work Order queries
  def work_orders_for(project_or_user)
  def filter_work_orders_by_date(query, params)

  # Run queries
  def runs_for(project_or_user)
  def filter_runs_by_date(query, params)

  # Step queries
  def steps_for(user)
  def last_step_for_job(job)
  def any_step()
  def steps_with_reason(query, reason)
  def last_successful_step_for_job(job)

  # Log Line queries
  def log_lines_for(user)
  def filter_log_lines(query, params)
end

1.4 Maintain Backwards Compatibility

Create facade module lib/lightning/invocation.ex (temporary):

defmodule Lightning.Invocation do
  @moduledoc false
  @moduledoc deprecated: """
  This module has been split into Lightning.Dataclips and Lightning.WorkOrders.
  It will be removed in a future version.
  """

  # Dataclip function delegates
  defdelegate list_dataclips(), to: Lightning.Dataclips
  defdelegate get_dataclip!(id), to: Lightning.Dataclips
  # ... all other dataclip functions

  # Step function delegates
  defdelegate get_step!(id), to: Lightning.WorkOrders
  defdelegate list_steps_for_project(project, params \\ %{}), to: Lightning.WorkOrders
  # ... all other step functions

  # Work order function delegates
  defdelegate search_workorders(project), to: Lightning.WorkOrders
  # ... all other work order functions
end

Create schema aliases (temporary):

# lib/lightning/invocation/dataclip.ex
defmodule Lightning.Invocation.Dataclip do
  @moduledoc false
  @deprecated "Use Lightning.Dataclips.Dataclip instead"
  use Lightning.Schema

  # Re-export from new location
  defdelegate __struct__(), to: Lightning.Dataclips.Dataclip
  defdelegate __struct__(kv), to: Lightning.Dataclips.Dataclip
  # Mirror all schema properties
end

# Similar for Step, LogLine, RunStep

Create lib/lightning/runs.ex facade (temporary):

defmodule Lightning.Runs do
  @moduledoc false
  @moduledoc deprecated: """
  This module has been merged into Lightning.WorkOrders.
  It will be removed in a future version.
  """

  defdelegate enqueue(run), to: Lightning.WorkOrders
  defdelegate get(id, opts \\ []), to: Lightning.WorkOrders
  defdelegate start_run(run, params \\ %{}), to: Lightning.WorkOrders
  # ... all other run functions
end

Phase 2: Update All Call Sites

Update all 51+ files that reference Lightning.Invocation or Lightning.Runs.

2.1 Update Module Aliases

Find and replace patterns:

# Old → New
alias Lightning.Invocation.Dataclip → alias Lightning.Dataclips.Dataclip
alias Lightning.Invocation.Step → alias Lightning.WorkOrders.Step
alias Lightning.Invocation.LogLine → alias Lightning.WorkOrders.LogLine
alias Lightning.Invocation.RunStep → alias Lightning.WorkOrders.RunStep
alias Lightning.Invocation → alias Lightning.Dataclips # or Lightning.WorkOrders
alias Lightning.Runs → alias Lightning.WorkOrders
alias Lightning.Run → alias Lightning.WorkOrders.Run

2.2 Update Function Calls

Dataclip operations:

# Old → New
Lightning.Invocation.get_dataclip!(id) → Lightning.Dataclips.get!(id)
Lightning.Invocation.list_dataclips_for_job(job, opts) → Lightning.Dataclips.list_for_job(job, opts)
Lightning.Invocation.create_dataclip(attrs) → Lightning.Dataclips.create(attrs)
Lightning.Invocation.Query.wipe_dataclips() → Lightning.Dataclips.Query.wipe()

Step operations:

# Old → New
Lightning.Invocation.get_step!(id) → Lightning.WorkOrders.get_step!(id)
Lightning.Invocation.get_first_step_for_run_and_job(run_id, job_id) → Lightning.WorkOrders.get_first_step_for_run_and_job(run_id, job_id)
Lightning.Invocation.Query.last_successful_step_for_job(job) → Lightning.WorkOrders.Query.last_successful_step_for_job(job)

Work order operations:

# Old → New
Lightning.Invocation.search_workorders(project, params) → Lightning.WorkOrders.search(project, params)
Lightning.Invocation.export_workorders(project, user, params) → Lightning.WorkOrders.export(project, user, params)
Lightning.Invocation.Query.work_orders_for(project) → Lightning.WorkOrders.Query.for_project(project)

Run operations:

# Old → New
Lightning.Runs.get(id, opts) → Lightning.WorkOrders.get_run(id, opts)
Lightning.Runs.start_run(run, params) → Lightning.WorkOrders.start_run(run, params)
Lightning.Runs.append_run_log(run, params) → Lightning.WorkOrders.append_log(run, params)
Lightning.Runs.Query.runs_for(project) → Lightning.WorkOrders.Query.runs_for(project)

2.3 Update Files by Category

Core Contexts (High Priority):

lib/lightning/work_orders.ex - Already being modified
lib/lightning/projects.ex - Uses dataclip wiping
lib/lightning/workflows/scheduler.ex - Uses last successful step
lib/lightning/credentials/resolver.ex - May reference schemas
lib/lightning/dashboard_stats.ex - Queries work orders/runs

Web Controllers: 6. lib/lightning_web/controllers/dataclip_controller.ex 7. lib/lightning_web/controllers/api/work_orders_controller.ex 8. lib/lightning_web/controllers/api/run_controller.ex 9. lib/lightning_web/controllers/api/log_lines_controller.ex

LiveViews: 10. lib/lightning_web/live/workflow_live/edit.ex - Heavy Invocation usage 11. lib/lightning_web/live/workflow_live/new_manual_run.ex 12. lib/lightning_web/live/run_live/index.ex 13. lib/lightning_web/live/run_live/components.ex 14. lib/lightning_web/live/dataclip_live/show.ex 15. lib/lightning_web/live/dataclip_live/form_component.ex

Background Workers: 16. lib/lightning/workorders/export_worker.ex 17. lib/lightning/digest_email_worker.ex

Remaining Files:

All other files from grep results (utilities, policies, etc.)

2.4 Update Tests

Test file moves:

test/lightning/invocation_test.exs → test/lightning/work_orders_test.exs (merge)
test/lightning/invocation/query_test.exs → test/lightning/work_orders/query_test.exs (merge)
test/lightning/invocation/dataclip_test.exs → test/lightning/dataclips/dataclip_test.exs
test/lightning/invocation/step_test.exs → test/lightning/work_orders/step_test.exs
test/lightning/invocation/log_line_test.exs → test/lightning/work_orders/log_line_test.exs
test/lightning/runs_test.exs → test/lightning/work_orders_test.exs (merge)

Update test factories:

# test/support/factories.ex
# Old → New
alias Lightning.Invocation.Dataclip → alias Lightning.Dataclips.Dataclip
alias Lightning.Invocation.Step → alias Lightning.WorkOrders.Step
alias Lightning.Run → alias Lightning.WorkOrders.Run

Update fixtures:

test/support/fixtures/invocation_fixtures.ex → Split into:
  - test/support/fixtures/dataclip_fixtures.ex
  - test/support/fixtures/work_order_fixtures.ex

Phase 3: Remove Lightning.Invocation & Lightning.Runs

3.1 Remove Facade Modules

Delete lib/lightning/invocation.ex
Delete lib/lightning/invocation/ directory
Delete lib/lightning/runs.ex (merge complete into WorkOrders)
Delete lib/lightning/runs/ directory (move all to work_orders/)

3.2 Verify Removal

Check for any remaining references:

# Should return no results
grep -r "Lightning\.Invocation" lib/
grep -r "Lightning\.Runs" lib/
grep -r "alias Lightning\.Invocation" lib/
grep -r "import Lightning\.Invocation" lib/

# Check test files too
grep -r "Lightning\.Invocation" test/
grep -r "Lightning\.Runs" test/

3.3 Update Documentation

Update CHANGELOG.md with breaking changes
Update any developer documentation
Update architectural diagrams if they exist
Update CLAUDE.md context file

Phase 4: Testing & Validation

4.1 Test Suite Validation

# Run full test suite
mix test

# Run specific test suites
mix test test/lightning/work_orders*
mix test test/lightning/dataclips*
mix test test/lightning_web/

# Run code quality checks
mix format --check-formatted
mix dialyzer
mix credo --strict --all
mix sobelow

# Run with coverage
mix coveralls.html

4.2 Integration Testing

# E2E tests
cd assets
npm run test:e2e

# Manual testing checklist:
# - Create work order via webhook
# - Create manual run
# - Retry a run
# - Search work orders with filters
# - Export work orders
# - View dataclips
# - Rename a dataclip
# - View run logs
# - Check cron-triggered workflows

Database Considerations

No database migrations required!

All database tables remain unchanged:

work_orders table - unchanged
runs table - unchanged
steps table - unchanged
run_steps table - unchanged
dataclips table - unchanged
log_lines table - unchanged

Only Elixir module organization changes.

API Compatibility

Public API Changes

If you have public API clients, these endpoints may be affected:

Potentially affected endpoints:

GET /api/work_orders - May have internal changes
GET /api/runs - May have internal changes
GET /api/dataclips - May have internal changes

Recommendation:

Keep API surface unchanged
Only change internal implementation
Version API if needed for major changes

Risks & Mitigation

Risk 1: Circular Dependencies

Risk: WorkOrders depends on Dataclips, but Dataclips queries may reference WorkOrders

Mitigation:

Keep Dataclips truly independent
Dataclips should not import WorkOrders
Use Ecto associations to load relationships
If needed, put cross-context queries in WorkOrders, not Dataclips

Risk 2: Large File Size

Risk: lib/lightning/work_orders.ex becomes too large (1500+ lines)

Mitigation:

Split into sub-modules:
- lib/lightning/work_orders/runs.ex - Run operations
- lib/lightning/work_orders/steps.ex - Step operations
- lib/lightning/work_orders/search.ex - Search operations
- Keep work_orders.ex as coordinator/facade
Or keep consolidated but well-organized with clear sections

Risk 3: Test Failures

Risk: Tests fail during migration due to missed references

Mitigation:

Use facade pattern in Phase 1 to catch issues early
Run test suite after each file update
Use mix test --failed to focus on broken tests
Keep a checklist of updated files

Risk 4: Performance Regression

Risk: Query reorganization causes performance issues

Mitigation:

Keep query logic identical during move
Run benchmarks before and after
Use EXPLAIN ANALYZE on key queries
Monitor production performance after deploy

Risk 5: Merge Conflicts

Risk: Active development causes merge conflicts during refactor

Mitigation:

Communicate refactor timeline to team
Consider feature freeze for affected modules
Break refactor into smaller, reviewable PRs
Use feature flag if needed for gradual rollout

Success Criteria

[ ] All tests pass (mix test)
[ ] No deprecation warnings
[ ] Dialyzer passes with no errors
[ ] Credo passes (mix credo --strict --all)
[ ] E2E tests pass
[ ] No references to Lightning.Invocation in codebase
[ ] No references to old Lightning.Runs in codebase
[ ] Documentation updated
[ ] CHANGELOG.md updated
[ ] Code review completed
[ ] Staging deployment successful
[ ] Production deployment successful

Benefits

Clear Domain Model: WorkOrders → Runs → Steps hierarchy is explicit
Single Responsibility: WorkOrders context owns entire execution lifecycle
Better Discoverability: Functions are in logical locations
Reduced Cognitive Load: No more "what's Invocation?"
Improved Maintainability: Execution logic consolidated in one place
Aligned with CTO Vision: Follows stated architectural direction

Open Questions

Should we split work_orders.ex into sub-modules (runs.ex, steps.ex, search.ex) or keep consolidated?
Should we rename some functions for consistency (e.g., get_step! → get_step! or keep as-is)?
Do we need a deprecation period with warnings, or can we do a clean break?
Should this be one large PR or multiple smaller PRs?
Do we need to update any public API versioning?

Next Steps

Review this plan
Answer open questions
Begin implementation
Review and test changes incrementally
Deploy when all phases complete successfully

References

Original discussion: CTO request to move models under WorkOrder
Affected files: 51+ files importing Lightning.Invocation
Related contexts: Lightning.WorkOrders, Lightning.Runs, Lightning.Projects

Oct 19 '25 12:10 taylordowns2000

Refactor namespacing to better fit our current design

Details

Implementation notes

Release notes

User acceptance criteria

Refactor Plan: Remove Lightning.Invocation Module

Executive Summary

Goals

Domain Model After Refactor

Rationale

Why move Run, Step, RunStep under WorkOrders?

Why keep Dataclips separate?

File Structure Changes

Current Structure

Target Structure

Detailed Migration Plan

Phase 1: Create New Structure (No Breaking Changes)

1.1 Create Lightning.Dataclips Context

1.2 Move Execution Schemas Under WorkOrders

1.3 Consolidate WorkOrders Functions

1.4 Maintain Backwards Compatibility

Phase 2: Update All Call Sites

2.1 Update Module Aliases

2.2 Update Function Calls

2.3 Update Files by Category

2.4 Update Tests

Phase 3: Remove Lightning.Invocation & Lightning.Runs

3.1 Remove Facade Modules

3.2 Verify Removal

3.3 Update Documentation

Phase 4: Testing & Validation

4.1 Test Suite Validation

4.2 Integration Testing

Database Considerations

API Compatibility

Public API Changes

Risks & Mitigation

Risk 1: Circular Dependencies

Risk 2: Large File Size

Risk 3: Test Failures

Risk 4: Performance Regression

Risk 5: Merge Conflicts

Success Criteria

Benefits

Open Questions

Next Steps

References