[Feature]: Allow running services and models without a gateway

[peterschmidt85]

Use case:

• As a user, I want to run a service/model and make it accessible to my project team.
• I don't want to set up a domain.
• I don't want to make the endpoint public.

Suggestion:

• In addition to the existing ability to create "managed" gateways (public IP address, custom domain), allow the dstack server to handle service/model traffic directly.

Motivation:

• So far, services have primarily been used for development, but the need to set up a domain and create a public endpoint poses a significant barrier. If the dstack server could run services/models out of the box, it would greatly simplify usage for both individuals and companies.

[peterschmidt85]

Another reason why this issue is important is that in our documentation, we often use tasks for deploying models because it doesn’t require creating a gateway, which is also not supported for all backends. However, when a model is running as a task, the user cannot use the UI to interact with the model.

[jvstme]

Implementation progress

• [ ] Python-based reverse proxy for services
  • [x] HTTP proxying
  • [x] Request/response streaming
  • [ ] Websocket proxying
  • [x] Proxying to multiple service replicas
  • [x] Auth
  • [ ] Configuring whether path prefix is stripped
• [x] Running services without a gateway
• [x] OpenAI endpoint
• [x] #1954
• [x] Configuring which gateway to use for the run
• [ ] Collecting stats for replicas autoscaling
• [x] Using the new proxy implementation on gateways
  • [x] Nginx- and certbot- related logic
  • [x] Installation and updates

Usage instructions for the current version

1. Make sure your project doesn't have a default gateway or set gateway: false in your service configuration.
2. Run the service.
3. Your service is now available at <dstack-server-base-url>/proxy/services/<project-name>/<run-name>/. If the service defines a model mapping, the OpenAI-compatible API is available at <dstack-server-base-url>/proxy/models/<project-name>/

[peterschmidt85]

Plus, ensure the UI can leverage the built-in proxy

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[jvstme]

Gateways have become optional in dstack 0.18.23, so we will close this issue. The remaining technical debt has been moved to separate issues to allow for more granular prioritization:

• #2252
• #2251
• #2250
• #2238
• #2237
• #2232
• #2231