Allow OPTIONS Method on Http Server and add Cors headers.

[kseyhan]

Hello, unfortunately i cant connect to the server with my frontend.

when running the server as follows: python -m sglang.launch_server --model-path mistralai/Mistral-7B-Instruct-v0.2 --port 30000

the options request fails with a 405:

curl "http://localhost:30000/v1/chat/completions" -X OPTIONS -H "Accept: */*" -H "Access-Control-Request-Method: POST"   -H "Origin: http://localhost:4200" -i
HTTP/1.1 405 Method Not Allowed
date: Thu, 28 Mar 2024 01:40:12 GMT
server: uvicorn
allow: POST
content-length: 31
content-type: application/json

{"detail":"Method Not Allowed"}

expected is something like this:

curl "http://localhost:8000/v1/chat/completions" -X OPTIONS -H "Accept: */*" -H "Access-Control-Request-Method: POST"   -H "Origin: http://localhost:4200"

HTTP/1.1 200 OK
date: Thu, 28 Mar 2024 01:38:28 GMT
server: uvicorn
vary: Origin
access-control-allow-methods: OPTIONS, POST
access-control-max-age: 600
access-control-allow-credentials: true
access-control-allow-origin: http://localhost:4200
content-length: 2
content-type: text/plain; charset=utf-8

[tom-doerr]

Could you send the server log? I think I got that error when it didn't finish loading the model.

[kseyhan]

the server is full loaded and the docs endpoint is accessible aswell:

python -m sglang.launch_server --model-path mistralai/Mistral-7B-Instruct-v0.2 --port 30000
Rank 0: load weight begin.
INFO 03-28 06:17:14 weight_utils.py:163] Using model weights format ['*.safetensors']
Rank 0: load weight end.
Rank 0: max_total_num_token=56507, max_prefill_num_token=32768, context_len=32768,
disable_radix_cache=False, enable_flashinfer=False, disable_regex_jump_forward=False, disable_disk_cache=False, attention_reduce_in_fp32=False
INFO:     Started server process [71216]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:30000 (Press CTRL+C to quit)
INFO:     127.0.0.1:57398 - "GET /get_model_info HTTP/1.1" 200 OK
new fill batch. #seq: 1. #cached_token: 0. #new_token: 9. #remaining_req: 0. #running_req: 0. tree_cache_hit_rate: 0.00%.
INFO:     127.0.0.1:57410 - "POST /generate HTTP/1.1" 200 OK
INFO:     127.0.0.1:41348 - "OPTIONS /v1/chat/completions HTTP/1.1" 405 Method Not Allowed
INFO:     127.0.0.1:33136 - "GET /docs HTTP/1.1" 200 OK

you can simply reproduce the error by fireing the curl request. did you try that?