Where should I place daily_pv.h5 file so that it can be found?

[lyenliang]

Hi, I just tried running rdagent fin_factor command.

This command always gives me FileNotFoundError: File daily_pv.h5 does not exist error.

I'm wondering where I should put this file?

I found that this file exists in RD-Agent/git_ignore_folder/factor_implementation_source_data/daily_pv.h5.

Why can't rdagent fin_factor detect this file automatically?

Here's the complete error message:

Role:user
Content: --------------Factor information:---------------
factor_name: Volume-Price Trend Factor
factor_description: This factor calculates the cumulative product of daily volume and the percentage change in closing price over a 20-day window. It aims to capture market momentum and investor sentiment by analyzing how changes in trading volume correlate with price movements.
factor_formulation: \text{Volume-Price Trend Factor}_{t} = \sum_{i=t-19}^{t} \left( V_{i} \times \frac{P_{i} - P_{i-1}}{P_{i-1}} \right)
variables: {'V_i': 'Trading volume on day i.', 'P_i': 'Closing price on day i.', 'P_{i-1}': 'Closing price on the previous day (i-1).'}
--------------Execution feedback:---------------
Traceback (most recent call last):
  File "/path/to/factor.py", line 23, in <module>
    main()
  File "/path/to/factor.py", line 16, in main
    df = pd.read_hdf('daily_pv.h5', key='data')
  File "/path/to/site-packages/pandas/io/pytables.py", line 424, in read_hdf
    raise FileNotFoundError(f"File {path_or_buf} does not exist")
FileNotFoundError: File daily_pv.h5 does not exist

Expected output file not found.

[lyenliang]

I've tried copying RD-Agent/git_ignore_folder/factor_implementation_source_data/daily_pv.h5 to RD-Agent/daily_pv.h5. Then I ran rdagent fin_factor under RD-Agent, but the problem still exists.

[lyenliang]

I found that factor.py is located under RD-Agent/git_ignore_folder/RD-Agent_workspace/1dbe60140382470691a25360af967948/, which looks like a dynamically generated folder. How can I create daily_pv.h5 in this folder?

[TPLin22]

Hi, You can try copying it to RD-Agent/git_ignore_folder/factor_implementation_source_data_debug/daily_pv.h5 and see if the issue is resolved."

[lyenliang]

Hi, @TPLin22 thanks for you reply. I found that daily_pv.h5 already existed in RD-Agent/git_ignore_folder/factor_implementation_source_data_debug/ when I ran rdagent fin_factor command.

I mean I had 2 daily_pv.h5 files. One is in RD-Agent/git_ignore_folder/factor_implementation_source_data_debug/, the other is in RD-Agent/git_ignore_folder/factor_implementation_source_data/.

So having RD-Agent/git_ignore_folder/factor_implementation_source_data_debug/daily_pv.h5 does not solve the problem.

[superafu]

Hi, @TPLin22 thanks for you reply. I found that daily_pv.h5 already existed in RD-Agent/git_ignore_folder/factor_implementation_source_data_debug/ when I ran rdagent fin_factor command.

I mean I had 2 daily_pv.h5 files. One is in RD-Agent/git_ignore_folder/factor_implementation_source_data_debug/, the other is in RD-Agent/git_ignore_folder/factor_implementation_source_data/.

So having RD-Agent/git_ignore_folder/factor_implementation_source_data_debug/daily_pv.h5 does not solve the problem.

Hi where did you find the daily_pv.h5 file? I met the same problem and can't find the .h5 file.

[inevity]

not use your market file, just let the rdagent to download

[weifw99]

I also encountered the same problem， The file exists. RD-Agent/git_ignore_folder/factor_implementation_source_data_debug/daily_pv.h5 (1.3M), RD-Agent/git_ignore_folder/factor_implementation_source_data/daily_pv.h5 (203M)

However, in the generated program, the path is the current directory df = pd.read_hdf('daily_pv.h5', key='data')

Error message：

=====Code to the former implementation=====

File Path: factor.py

import pandas as pd

def calculate_VolumeZScoreDeviation_10():
    df = pd.read_hdf("daily_pv.h5", key="data")
    df = df.sort_index()
    df_reset = df.reset_index()
    df_reset = df_reset.sort_values(by=["instrument", "datetime"])
    grouped = df_reset.groupby("instrument")
    df_reset['rolling_mean_10'] = grouped['volume'].transform(lambda x: x.rolling(10).mean())
    df_reset['rolling_std_10'] = grouped['volume'].transform(lambda x: x.rolling(10).std())
    df_reset['VolumeZScoreDeviation_10'] = (df_reset['volume'] - df_reset['rolling_mean_10']) / df_reset['rolling_std_10']
    result_df = df_reset.set_index(["datetime", "instrument"])[['VolumeZScoreDeviation_10']]
    result_df.to_hdf('result.h5', key='data')

if __name__ == "__main__":
    calculate_VolumeZScoreDeviation_10()

=====Feedback to the former implementation===== ------------------Execution Feedback------------------ Traceback (most recent call last): File "/path/to/factor.py", line 16, in calculate_VolumeZScoreDeviation_10() File "/path/to/factor.py", line 4, in calculate_VolumeZScoreDeviation_10 df = pd.read_hdf("daily_pv.h5", key="data") File "/path/to/site-packages/pandas/io/pytables.py", line 424, in read_hdf raise FileNotFoundError(f"File {path_or_buf} does not exist") FileNotFoundError: File daily_pv.h5 does not exist

Expected output file not found. ------------------Shape Feedback------------------

------------------Code Feedback------------------

[ChenhuaFan]

same issue

[EricChengg]

same issue

[qq2100803]

same issue

[fafuzuoluo]

same issue

[fafuzuoluo]

生成的分析代码，读取文件路径有问题

[multicollinearity]

same issue here. is there a fix?

[xinkui]

这个是没有对mac系统文件符号链接支持，可以先修改源码：/opt/anaconda3/envs/rdagent4qlib/lib/python3.10/site-packages/rdagent/core/experiment.py 这个里面的这个方法替换成以下（添加对mac的支持）： @staticmethod def link_all_files_in_folder_to_workspace(data_path: Path, workspace_path: Path) -> None: data_path = Path(data_path).absolute() # in case of relative path that will be invalid when we change cwd. workspace_path = Path(workspace_path) for data_file_path in data_path.iterdir(): workspace_data_file_path = workspace_path / data_file_path.name if workspace_data_file_path.exists(): workspace_data_file_path.unlink() if platform.system() == "Linux": os.symlink(data_file_path, workspace_data_file_path) elif platform.system() == "Windows": os.link(data_file_path, workspace_data_file_path) elif platform.system() == "Darwin": # macOS support os.symlink(data_file_path, workspace_data_file_path)

希望开发人员fix它，下个版本可以发布。