Error reports when loading model checkpoint

[JackeyLee007]

When the following code executes,

import timesfm
import os
import numpy as np


tfm = timesfm.TimesFm(
    context_len=64,
    horizon_len=32,
    input_patch_len=32,
    output_patch_len=128,
    num_layers=20,
    model_dims=1280,
    backend='cpu',
)
tfm.load_from_checkpoint(repo_id="google/timesfm-1.0-200m")

forecast_input = [
    np.sin(np.linspace(0, 20, 100)),
    np.sin(np.linspace(0, 20, 200)),
    np.sin(np.linspace(0, 20, 400)),
]
frequency_input = [0, 1, 2]

print('before predicate')
point_forecast, experimental_quantile_forecast = tfm.forecast(
    forecast_input,
    freq=frequency_input,
)

print('after predicate')
print(point_forecast)

It reports the following errors

Restoring checkpoint from /root/.cache/huggingface/hub/models--google--timesfm-1.0-200m/snapshots/8775f7531211ac864b739fe776b0b255c277e2be/checkpoints.
WARNING:absl:No registered CheckpointArgs found for handler type: <class 'paxml.checkpoints.FlaxCheckpointHandler'>
WARNING:absl:Configured `CheckpointManager` using deprecated legacy API. Please follow the instructions at https://orbax.readthedocs.io/en/latest/api_refactor.html to migrate by May 1st, 2024.
WARNING:absl:train_state_unpadded_shape_dtype_struct is not provided. We assume `train_state` is unpadded.
Killed
(timfm_env02) root@instance:~/opensrc# /root/miniconda3/envs/timfm_env02/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

[king571834]

I have the same problem.

[JackeyLee007]

@siriuz42 Take a look at this issue. Would you please help us out?

[siriuz42]

My current guess is that this is a RAM issue - I've seen this issue once when I was loading the model on a 8G RAM machine, and resolved it by going up to 16G.

[king571834]

我目前的猜測是這是一個 RAM 問題 - 當我在 8G RAM 機器上加載模型時，我曾經遇到過這個問題，並通過升級到 16G 來解決它。

But my RAM is 48G also has the same problem

[JackeyLee007]

My current guess is that this is a RAM issue - I've seen this issue once when I was loading the model on a 8G RAM machine, and resolved it by going up to 16G.

@siriuz42 Thanks. It solved my problem. I raised my RAM to 64G, then it works well so far.

[siriuz42]

@king571834 Could you paste your error message / stack trace if there's any?