Crash due: SQLite objects created in a thread can only be used in that same thread

[FFAMax]

Found instability:

Error processing tensor for shard Shard(model_id='mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated', start_layer=0, end_layer=15,
n_layers=32): SQLite objects created in a thread can only be used in that same thread. The object was created in thread id
139810278716992 and this is thread id 139809641199168.
Traceback (most recent call last):
  File "/home/ffamax/exo/exo/orchestration/standard_node.py", line 239, in _process_tensor
    result, inference_state, is_finished = await self.inference_engine.infer_tensor(request_id, shard, tensor,
inference_state=inference_state)
  File "/home/ffamax/exo/exo/inference/tinygrad/inference.py", line 76, in infer_tensor
    await self.ensure_shard(shard)
  File "/home/ffamax/exo/exo/inference/tinygrad/inference.py", line 97, in ensure_shard
    self.model = await asyncio.get_event_loop().run_in_executor(self.executor, build_transformer, model_path, shard, "8B" if "8b"
in shard.model_id.lower() else "70B")
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/ffamax/exo/exo/inference/tinygrad/inference.py", line 48, in build_transformer
    load_state_dict(model, weights, strict=False, consume=False)  # consume=True
  File "/home/ffamax/exo/.venv/lib/python3.10/site-packages/tinygrad/nn/state.py", line 129, in load_state_dict
    else: v.replace(state_dict[k].to(v.device)).realize()
  File "/home/ffamax/exo/.venv/lib/python3.10/site-packages/tinygrad/tensor.py", line 3475, in _wrapper
    if _METADATA.get() is not None: return fn(*args, **kwargs)
  File "/home/ffamax/exo/.venv/lib/python3.10/site-packages/tinygrad/tensor.py", line 213, in realize
    run_schedule(*self.schedule_with_vars(*lst), do_update_stats=do_update_stats)
  File "/home/ffamax/exo/.venv/lib/python3.10/site-packages/tinygrad/engine/realize.py", line 222, in run_schedule
    for ei in lower_schedule(schedule):
  File "/home/ffamax/exo/.venv/lib/python3.10/site-packages/tinygrad/engine/realize.py", line 215, in lower_schedule
    raise e
  File "/home/ffamax/exo/.venv/lib/python3.10/site-packages/tinygrad/engine/realize.py", line 209, in lower_schedule
    try: yield lower_schedule_item(si)
  File "/home/ffamax/exo/.venv/lib/python3.10/site-packages/tinygrad/engine/realize.py", line 193, in lower_schedule_item
    runner = get_runner(si.outputs[0].device, si.ast)
  File "/home/ffamax/exo/.venv/lib/python3.10/site-packages/tinygrad/engine/realize.py", line 162, in get_runner
    method_cache[ckey] = method_cache[bkey] = ret = CompiledRunner(replace(prg, dname=dname))
  File "/home/ffamax/exo/.venv/lib/python3.10/site-packages/tinygrad/engine/realize.py", line 84, in __init__
    self.lib:bytes = precompiled if precompiled is not None else Device[p.dname].compiler.compile_cached(p.src)
  File "/home/ffamax/exo/.venv/lib/python3.10/site-packages/tinygrad/device.py", line 181, in compile_cached
    if self.cachekey is None or (lib := diskcache_get(self.cachekey, src)) is None:
  File "/home/ffamax/exo/.venv/lib/python3.10/site-packages/tinygrad/helpers.py", line 234, in diskcache_get
    cur = conn.cursor()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread
id 139810278716992 and this is thread id 139809641199168.