inference icon indicating copy to clipboard operation
inference copied to clipboard

Published 20 hours ago • verified publisher icon xorbitsai

如何在容器中使用xinference-worker进行部署

Open ZJU-lishuang opened this issue 6 months ago • 17 comments

通过容器启动supervisor

docker run -v ./xinfer_supervisor:/tmp/xinference --name xinfer_supervisor \
-e XINFERENCE_HOME=/tmp/xinference -e XINFERENCE_MODEL_SRC=modelscope \
-p 9997:9997 -p 9996:9996 xprobe/xinference:v0.13.2 \
xinference-supervisor -H 0.0.0.0 -p 9997 --supervisor-port 9996 --log-level debug

在另一台服务器上，在容器中启动worker失败。 指令

docker run -v ./xinfer_worker:/tmp/xinference --name xinfer_worker \
-e XINFERENCE_HOME=/tmp/xinference -e XINFERENCE_MODEL_SRC=modelscope \
--net=host --gpus '"device=6,7"' xprobe/xinference:v0.13.2 \
xinference-worker -e "http://${supervisor_host}:9997" -H 0.0.0.0 --worker-port 9995 --metrics-exporter-port 9994

报错

Traceback (most recent call last):
  File "/usr/local/bin/xinference-worker", line 8, in <module>
    sys.exit(worker())
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/xinference/deploy/cmdline.py", line 354, in worker
    main(
  File "/usr/local/lib/python3.10/dist-packages/xinference/deploy/worker.py", line 94, in main
    loop.run_until_complete(task)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/dist-packages/xinference/deploy/worker.py", line 65, in _start_worker
    await start_worker_components(
  File "/usr/local/lib/python3.10/dist-packages/xinference/deploy/worker.py", line 43, in start_worker_components
    await xo.create_actor(
  File "/usr/local/lib/python3.10/dist-packages/xoscar/api.py", line 78, in create_actor
    return await ctx.create_actor(actor_cls, *args, uid=uid, address=address, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 143, in create_actor
    return self._process_result_message(result)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 102, in _process_result_message
    raise message.as_instanceof_cause()
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/pool.py", line 595, in create_actor
    await self._run_coro(message.message_id, actor.__post_create__())
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/pool.py", line 367, in _run_coro
    return await coro
  File "/usr/local/lib/python3.10/dist-packages/xinference/core/worker.py", line 185, in __post_create__
    ] = await xo.actor_ref(
  File "/usr/local/lib/python3.10/dist-packages/xoscar/api.py", line 125, in actor_ref
    return await ctx.actor_ref(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 197, in actor_ref
    future = await self._call(actor_ref.address, message, wait=False)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 77, in _call
    return await self._caller.call(
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/core.py", line 181, in call
    client = await self.get_client(router, dest_address)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/core.py", line 68, in get_client
    client = await router.get_client(dest_address, from_who=self)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/router.py", line 143, in get_client
    client = await self._create_client(client_type, address, **kw)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/router.py", line 157, in _create_client
    return await client_type.connect(address, local_address=local_address, **kw)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/communication/socket.py", line 255, in connect
    (reader, writer) = await asyncio.open_connection(host=host, port=port, **kwargs)
  File "/usr/lib/python3.10/asyncio/streams.py", line 48, in open_connection
    transport, _ = await loop.create_connection(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1076, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1060, in create_connection
    sock = await self._connect_sock(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 969, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 501, in sock_connect
    return await fut
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 541, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [address=0.0.0.0:9995, pid=1890] [Errno 111] Connect call failed ('0.0.0.0', 9996)

指令

docker run -v ./xinfer_worker:/tmp/xinference --name xinfer_worker \
-e XINFERENCE_HOME=/tmp/xinference -e XINFERENCE_MODEL_SRC=modelscope \
--net=host --gpus '"device=6,7"' xprobe/xinference:v0.13.2 \
xinference-worker -e "http://${supervisor_host}:9997" -H "${worker_host}" --worker-port 9995 --metrics-exporter-port 9994

报错

2024-07-25 07:29:40,967 xinference.core.worker 2026 INFO     Starting metrics export server at 0.0.0.0:9994
2024-07-25 07:29:40,968 xinference.core.worker 2026 INFO     Checking metrics export server...
2024-07-25 07:29:42,020 xinference.core.worker 2026 INFO     Metrics server is started at: http://0.0.0.0:9994
Traceback (most recent call last):
  File "/usr/local/bin/xinference-worker", line 8, in <module>
    sys.exit(worker())
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/xinference/deploy/cmdline.py", line 354, in worker
    main(
  File "/usr/local/lib/python3.10/dist-packages/xinference/deploy/worker.py", line 94, in main
    loop.run_until_complete(task)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/dist-packages/xinference/deploy/worker.py", line 65, in _start_worker
    await start_worker_components(
  File "/usr/local/lib/python3.10/dist-packages/xinference/deploy/worker.py", line 43, in start_worker_components
    await xo.create_actor(
  File "/usr/local/lib/python3.10/dist-packages/xoscar/api.py", line 78, in create_actor
    return await ctx.create_actor(actor_cls, *args, uid=uid, address=address, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 143, in create_actor
    return self._process_result_message(result)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 102, in _process_result_message
    raise message.as_instanceof_cause()
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/pool.py", line 595, in create_actor
    await self._run_coro(message.message_id, actor.__post_create__())
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/pool.py", line 367, in _run_coro
    return await coro
  File "/usr/local/lib/python3.10/dist-packages/xinference/core/worker.py", line 185, in __post_create__
    ] = await xo.actor_ref(
  File "/usr/local/lib/python3.10/dist-packages/xoscar/api.py", line 125, in actor_ref
    return await ctx.actor_ref(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 197, in actor_ref
    future = await self._call(actor_ref.address, message, wait=False)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/context.py", line 77, in _call
    return await self._caller.call(
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/core.py", line 181, in call
    client = await self.get_client(router, dest_address)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/core.py", line 68, in get_client
    client = await router.get_client(dest_address, from_who=self)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/router.py", line 143, in get_client
    client = await self._create_client(client_type, address, **kw)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/router.py", line 157, in _create_client
    return await client_type.connect(address, local_address=local_address, **kw)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/communication/socket.py", line 255, in connect
    (reader, writer) = await asyncio.open_connection(host=host, port=port, **kwargs)
  File "/usr/lib/python3.10/asyncio/streams.py", line 48, in open_connection
    transport, _ = await loop.create_connection(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1076, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1060, in create_connection
    sock = await self._connect_sock(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 969, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 501, in sock_connect
    return await fut
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 541, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [address="${worker_host}":9995, pid=2026] [Errno 111] Connect call failed ('0.0.0.0', 9996)

请问可以限制worker端口的使用吗，如下命令应该是怎么样的才能把worker跑起来。

docker run -v ./xinfer_worker:/tmp/xinference --name xinfer_worker \
-e XINFERENCE_HOME=/tmp/xinference -e XINFERENCE_MODEL_SRC=modelscope \
-p 9995:9995 -p 9994:9994 --gpus '"device=6,7"' xprobe/xinference:v0.13.2 \
xinference-worker -e "http://${supervisor_host}:9997" -H 0.0.0.0 --worker-port 9995 --metrics-exporter-port 9994

ZJU-lishuang avatar Jul 25 '24 07:07 ZJU-lishuang