KoboldAI-Client icon indicating copy to clipboard operation
KoboldAI-Client copied to clipboard

Published 20 hours ago • verified publisher icon KoboldAI

Erebus 20B - Can't get Meta Device Value due to it being N/A

Open coldReactive opened this issue 2 years ago • 5 comments 

INIT       | Searching  | GPU support
INIT       | Found      | GPU support
INIT       | Starting   | Transformers
INIT       | Info       | Final device configuration:
       DEVICE ID  |  LAYERS  |  DEVICE NAME
               0  |      11  |  NVIDIA GeForce RTX 3060
             N/A  |      11  |  (Disk cache)
             N/A  |      22  |  (CPU)
INIT       | Loading model tensors: 100%|##########| 664/664 [01:15<00:00,  8.79it/s]
Exception in thread Thread-24:
Traceback (most recent call last):
  File "B:\python\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "B:\python\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "B:\python\lib\site-packages\socketio\server.py", line 731, in _handle_event_internal
    r = server._trigger_event(data[0], namespace, sid, *data[1:])
  File "B:\python\lib\site-packages\socketio\server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "B:\python\lib\site-packages\flask_socketio\__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "B:\python\lib\site-packages\flask_socketio\__init__.py", line 826, in _handle_event
    ret = handler(*args)
  File "aiserver.py", line 466, in g
    return f(*a, **k)
  File "aiserver.py", line 3917, in get_message
    load_model(use_gpu=msg['use_gpu'], gpu_layers=msg['gpu_layers'], disk_layers=msg['disk_layers'], online_model=msg['online_model'])
  File "aiserver.py", line 2654, in load_model
    move_model_to_devices(model)
  File "aiserver.py", line 941, in move_model_to_devices
    accelerate.utils.set_module_tensor_to_device(model, key, target_dtype)
  File "B:\python\lib\site-packages\accelerate\utils\modeling.py", line 118, in set_module_tensor_to_device
    raise ValueError(f"{tensor_name} is on the meta device, we need a `value` to put in on {device}.")
ValueError: bias is on the meta device, we need a `value` to put in on torch.float16.

coldReactive avatar Dec 16 '22 19:12 coldReactive

Still happening today even after I did an update:

Runtime launching in B: drive mode
INIT       | Starting   | Flask
INIT       | OK         | Flask
INIT       | Starting   | Webserver
INIT       | Starting   | LUA bridge
INIT       | OK         | LUA bridge
INIT       | Starting   | LUA Scripts
INIT       | OK         | LUA Scripts
INIT       | OK         | Webserver
MESSAGE    | Webserver started! You may now connect with a browser at http://127.0.0.1:5000
INFO       | __main__:do_connect:3541 - Client connected!
INIT       | Searching  | GPU support
INIT       | Found      | GPU support
INIT       | Starting   | Transformers
INIT       | Info       | Final device configuration:
       DEVICE ID  |  LAYERS  |  DEVICE NAME
               0  |      11  |  NVIDIA GeForce RTX 3060
             N/A  |      11  |  (Disk cache)
             N/A  |      22  |  (CPU)
INIT       | Loading model tensors: 100%|##########| 664/664 [01:20<00:00,  8.25it/s]
Exception in thread Thread-13:
Traceback (most recent call last):
  File "B:\python\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "B:\python\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "B:\python\lib\site-packages\socketio\server.py", line 731, in _handle_event_internal
    r = server._trigger_event(data[0], namespace, sid, *data[1:])
  File "B:\python\lib\site-packages\socketio\server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "B:\python\lib\site-packages\flask_socketio\__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "B:\python\lib\site-packages\flask_socketio\__init__.py", line 826, in _handle_event
    ret = handler(*args)
  File "aiserver.py", line 466, in g
    return f(*a, **k)
  File "aiserver.py", line 3914, in get_message
    load_model(use_gpu=msg['use_gpu'], gpu_layers=msg['gpu_layers'], disk_layers=msg['disk_layers'], online_model=msg['online_model'])
  File "aiserver.py", line 2651, in load_model
    move_model_to_devices(model)
  File "aiserver.py", line 941, in move_model_to_devices
    accelerate.utils.set_module_tensor_to_device(model, key, target_dtype)
  File "B:\python\lib\site-packages\accelerate\utils\modeling.py", line 118, in set_module_tensor_to_device
    raise ValueError(f"{tensor_name} is on the meta device, we need a `value` to put in on {device}.")
ValueError: bias is on the meta device, we need a `value` to put in on torch.float16.

coldReactive avatar Dec 19 '22 02:12 coldReactive

Also, B: Drive doesn't get removed when the command line for the server terminates sometimes.

coldReactive avatar Dec 19 '22 22:12 coldReactive

Still happening even after I searched for a Main Update today:

Runtime launching in B: drive mode
INIT       | Starting   | Flask
INIT       | OK         | Flask
INIT       | Starting   | Webserver
INIT       | Starting   | LUA bridge
INIT       | OK         | LUA bridge
INIT       | Starting   | LUA Scripts
INIT       | OK         | LUA Scripts
INIT       | OK         | Webserver
MESSAGE    | Webserver started! You may now connect with a browser at http://127.0.0.1:5000
Traceback (most recent call last):
  File "B:\python\lib\site-packages\eventlet\hubs\selects.py", line 59, in wait
    listeners.get(fileno, hub.noop).cb(fileno)
  File "B:\python\lib\site-packages\eventlet\greenthread.py", line 221, in main
    result = function(*args, **kwargs)
  File "B:\python\lib\site-packages\eventlet\wsgi.py", line 837, in process_request
    proto.__init__(conn_state, self)
  File "B:\python\lib\site-packages\eventlet\wsgi.py", line 352, in __init__
    self.finish()
  File "B:\python\lib\site-packages\eventlet\wsgi.py", line 751, in finish
    BaseHTTPServer.BaseHTTPRequestHandler.finish(self)
  File "B:\python\lib\socketserver.py", line 811, in finish
    self.wfile.close()
  File "B:\python\lib\socket.py", line 687, in write
    return self._sock.send(b)
  File "B:\python\lib\site-packages\eventlet\greenio\base.py", line 401, in send
    return self._send_loop(self.fd.send, data, flags)
  File "B:\python\lib\site-packages\eventlet\greenio\base.py", line 388, in _send_loop
    return send_method(data, *args)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
Removing descriptor: 1408
INFO       | __main__:do_connect:3541 - Client connected!
INIT       | Searching  | GPU support
INIT       | Found      | GPU support
INIT       | Starting   | Transformers
INIT       | Info       | Final device configuration:
       DEVICE ID  |  LAYERS  |  DEVICE NAME
               0  |      11  |  NVIDIA GeForce RTX 3060
             N/A  |      11  |  (Disk cache)
             N/A  |      22  |  (CPU)
INIT       | Loading model tensors: 100%|##########| 664/664 [01:24<00:00,  7.86it/s]
Exception in thread Thread-13:
Traceback (most recent call last):
  File "B:\python\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "B:\python\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "B:\python\lib\site-packages\socketio\server.py", line 731, in _handle_event_internal
    r = server._trigger_event(data[0], namespace, sid, *data[1:])
  File "B:\python\lib\site-packages\socketio\server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "B:\python\lib\site-packages\flask_socketio\__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "B:\python\lib\site-packages\flask_socketio\__init__.py", line 826, in _handle_event
    ret = handler(*args)
  File "aiserver.py", line 466, in g
    return f(*a, **k)
  File "aiserver.py", line 3914, in get_message
    load_model(use_gpu=msg['use_gpu'], gpu_layers=msg['gpu_layers'], disk_layers=msg['disk_layers'], online_model=msg['online_model'])
  File "aiserver.py", line 2651, in load_model
    move_model_to_devices(model)
  File "aiserver.py", line 941, in move_model_to_devices
    accelerate.utils.set_module_tensor_to_device(model, key, target_dtype)
  File "B:\python\lib\site-packages\accelerate\utils\modeling.py", line 118, in set_module_tensor_to_device
    raise ValueError(f"{tensor_name} is on the meta device, we need a `value` to put in on {device}.")
ValueError: bias is on the meta device, we need a `value` to put in on torch.float16.
INFO       | __main__:do_connect:3541 - Client connected!

coldReactive avatar Jan 06 '23 01:01 coldReactive

Even after reloading into the browser interface (since the model is loaded) the model accepts no inputs whatsoever. Typing into the big textarea then clicking submit does nothing. Even though it told me to write up several sentences, which I did:

How to use this model

Erebus is designed to generate short stories and novels. Use the author's note to give it a certain genre to follow, use memory to give an overview of the story and use World Information to give specific details about the characters. To start off, give the AI an idea of what you are writing about by setting the scene. Give the AI around 10 sentences that make your story interesting to read. Introduce your character, describe the world, blow something up, or let the AI use its creative mind.

coldReactive avatar Jan 06 '23 01:01 coldReactive

Same crash here. Although the 32GB version of Erebus worked fine for me, it was just the 64GB version that produced this error.

koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/accelerate/utils/modeling.py", line 127, in set_module_tensor_to_device
    raise ValueError(f"{tensor_name} is on the meta device, we need a `value` to put in on {device}.")
ValueError: bias is on the meta device, we need a `value` to put in on torch.float16.

Skyedra avatar Feb 05 '23 09:02 Skyedra