Implement API for backends configuration via yaml

[r4victor]

Currently, dstack server provides API to create/update backends. This is supposed to be used by UIs/CLIs that provide alternative to ~/.dstack/server/config.yml. But implementing an interface on top of this API takes effort since it requires custom code for every backend type. The proposal is to introduce a simple backends configuration API that would reuse the same yaml logic that used for configuring via ~/.dstack/server/config.yml.

Introduce two new API endpoints:

1. POST /api/project/main/backends/get_config

   Returns a yaml string with a list of configured backends for the project in the same format as ~/.dstack/server/config.yml.

2. POST /api/project/main/backends/configure

   Accepts a yaml string with a list of backends to configure for the project in the same format as ~/.dstack/server/config.yml.

These API methods will only be available to project admins. To show configured backend types to all users, the existing /api/projects/{project_name}/get endpoint can be consulted.

Limitations:

• There won't be a way to upload file for GCP service account, Kubeconfig, etc. Their contents have to be specified directly. When configuring via ~/.dstack/server/config.yml, the dstack server can pick up those files by filepath, but it cannot pick up files from the machine that is used to access the UI. The upload has to be supported on the UI side.

[peterschmidt85]

I'd suggest that ideally, the API is per backend. This will allow the user to edit individual backends rather than updating all backends at once.

[r4victor]

After the discussion, we decided to allow configuring each backend separately. The new API will be structured just as the current one but accepting/returning config as yaml string.

The new endpoints are:

1. POST /api/project/{project_name}/backends/create_yaml

Accepts a yaml string with backend config. If backend exists, returns error.

2. POST /api/project/{project_name}/backends/update_yaml

Accepts a yaml string with backend config. If backend exists, overrides the config.

3. POST /api/project/{project_name}/backends/get_yaml

Returns a yaml string with backend config.

Backend config example:

type: aws
  regions: [eu-west-1, us-east-2, us-west-1, us-west-2, ca-central-1, eu-central-1]
  creds:
    type: access_key
    access_key: ACCESS_KEY
    secret_key: SECRET_KEY

The current json API will be preserved for now but may be deprecated in the future.