infra
infra copied to clipboard
Add peer-to-peer metrics tracker
This consolidates metrics into a single struct that does a few things:
- exports metrics to a file called
{pid}.json - watches for other files and reads their metrics.
- when handling incoming requests, check full host metrics
It also creates server.Limiter for checking starting and running limits.
[!NOTE] Introduces a shared-state manager that aggregates sandbox allocations across processes via PID JSON files and integrates a limiter to cap starting/running sandboxes, wiring both into server and service info paths.
- Shared State (peer-to-peer metrics):
- Add
internal/sharedstatewithManagerthat writes self allocations to{pid}.json, watches directory viafsnotify, and aggregatesAllocationsacross processes.- Integrate with
sandboxesmap viaSubscribe; exposeTotalAllocated()andTotalRunningCount().- Add tests in
internal/sharedstate/tracker_test.go.- Sandbox start limiting:
- Add
server.Limiterto enforce max running (viafeatureflags.MaxSandboxesPerNode) and max starting per node; replace semaphore logic inserverwithsandboxLimiterand error handling.- Service info metrics:
internal/service/service_info.go: switch from iterating localsandboxesto usingsharedstate.Managerfor CPU/memory/disk and running counts.- Config:
- Add
SHARED_STATE_DIRECTORY,SHARED_STATE_WRITE_INTERVAL,MAX_STARTING_INSTANCEStocfg.Config.- Wiring:
main.go: instantiate and runsharedstate.Manager; pass toInfoServiceandserver.New; createserver.NewLimiter.- Minor: rename Google Storage limiter variable for clarity.
- Dependencies:
- Add
github.com/fsnotify/fsnotifytogo.mod/go.sum.Written by Cursor Bugbot for commit 835ee09fd8a776787cd71b2d053c1197c9c4d975. This will update automatically on new commits. Configure here.
I'm going to pull a piece out to its own PR, and reopen when that's merged.