nos
nos copied to clipboard
autonomi-ai
⚡️ A fast and flexible PyTorch inference server that runs locally, on any cloud or AI HW.
Reduce bloat, move init, ready, id etc into the subclass. Right now we just have an inference runtime, but future releases might include runtimes for benchmarking, compilation etc.
If we're able to build checksums for layer-wise weights, we should be able to only download the diffs and speed up model downloads significantly. This is particularly helpful if you're...
Currently we create a large docker image (11GB) for the base gpu image
- Github workflow CI: Support platforms: Windows, MacOS for basic models (SD v2, CLIP)
- `nos serve -m stability-ai/stable-diffusion-v2`: Serve optimized `nos` model (blocking) - `nos serve -d stability-ai/stable-diffusion-v2`: Serve optimized `nos model (daemon/detached) - `nos serve -c deployment.yml`: Serve collection of models (blocking)...
Register models as part of the nos hub registry, with full build-time and runtime spec. ```python @hub.register( name="/detection2d-detr-resnet-50", build_spec=DevelopmentConfig( conda="autonomi-ai/nos-base-dev", resources=ResourceConfig(cpu=8, memory="8Gi", gpu=0.25, gpu_memory="4Gi"), # runtime resource ), runtime_spec=RuntimeConfig( conda="autonomi-ai/nos-base-runtime",...
Metadata
Owner
Metadata
⚡️ A fast and flexible PyTorch inference server that runs locally, on any cloud or AI HW.