[Feature]: Support `resources` with the `gateway` configuration type

[Stealthwriter]

If I send too many requests, like 2000 per second, the gateway can't handle that. Also, I have 9 services running. We need to either run multiple gateways or at least be able to increase instance resources.

Proposal:

Allow to specify resources for the gateway configuration type –just like for runs (dev-environment, task, and service) configurations.

[Stealthwriter]

skypilot has this already.

Customizing SkyServe controller resources You may want to customize the resources of the SkyServe controller for several reasons:

Use a lower-cost controller. (if you have a few services running)

Enforcing the controller to run on a specific location. This is particularly useful when you want the service endpoint within specific geographical region. (Default: cheapest location)

Changing the maximum number of services that can be run concurrently, which is the minimum number between 4x the vCPUs of the controller and the memory in GiB of the controller. (Default: 16)

Changing the disk_size of the controller to store more logs. (Default: 200GB)

To achieve the above, you can specify custom configs in ~/.sky/config.yaml with the following fields:

[Stealthwriter]

this issue is causing a bottle neck for users who have many services/users

[r4victor]

@Stealthwriter, dstack indeed uses too small gateway instances to handle such loads (e.g. "t2.micro" on AWS).

If you run dstack from source, you can change the hardcoded instance type and disk_size here:

https://github.com/dstackai/dstack/blob/f65402285db42888a358e6fb4e4ef9daeca43ec8/src/dstack/_internal/core/backends/aws/compute.py#L275-L277

Please try changing them and provision a new gateway and see if the problem persist. If it helps, we're going to prioritize making instance_type for gateway configurable.

[jvstme]

We need to be able to run multiple gateways

@Stealthwriter, it is possible to create different gateways for different services to spread the load — just create and apply multiple gateway configurations. Choosing a gateway for a service is awkward now but it can be done by setting the default gateway before running the service.

dstack gateway update --set-default gateway-1
dstack apply -f service-1.dstack.yml
dstack gateway update --set-default gateway-2
dstack apply -f service-2.dstack.yml

dstack will likely introduce a better interface for choosing between gateways as part of #1595.

Note that different gateways should have different domain names. Running multiple replicas of the same gateway with the same domain name is not supported yet.

[Stealthwriter]

good idea thanks

[naaa760]

Hello @Stealthwriter :) I want to contribute to this issue. Please assign this issue to me. Thank you!

[jvstme]

Hi @naaa760! Thank you for your interest in contributing to dstack. Assignment is not necessary, so please feel free to submit a PR if you know how to implement this. Note that this is a major feature that will require changes and testing in all dstack backends that support gateways: AWS, Azure, GCP, and Kubernetes.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.