dump_bin DumpDataUpdate mode append data error

[hbhuyt]

🐛 Bug Description

At first, I used Dump_bin's DumpDataAll mode to import data it worked fine. Part of the imported data is as follows df[df['instrument']=='SH600306'] Out[35]: instrument datetime $volume $factor $close 41691 SH600306 2024-04-23 1022018.0 0.281253 0.686257 41692 SH600306 2024-04-24 1372334.0 0.281253 0.652507 41693 SH600306 2024-04-25 951008.0 0.281253 0.618756 41694 SH600306 2024-04-26 1968818.0 0.281253 0.587818 41695 SH600306 2024-04-29 1532764.0 0.281253 0.559693

But when I append new data with DumpDataUpdate, there is an error. The original data is as follows dfraw.loc[(dfraw['date']>'2024-04-29'),['instrument','date','close']] Out[54]: instrument date close 4356 SH600306 2024-05-29 0.098438 4357 SH600306 2024-05-30 0.092813 4358 SH600306 2024-05-31 0.101251 4359 SH600306 2024-06-03 0.092813 4360 SH600306 2024-06-04 0.095626 4361 SH600306 2024-06-05 0.092813 4362 SH600306 2024-06-06 0.092813 4363 SH600306 2024-06-07 0.095626 4364 SH600306 2024-06-11 0.090001 4365 SH600306 2024-06-12 0.090001 4366 SH600306 2024-06-13 0.087188 4367 SH600306 2024-06-14 0.081563

Some of the imported data is shown below

dfnew[dfnew.instrument=='SH600306'] Out[8]: instrument datetime $volume $factor $close 10288 SH600306 2024-04-22 363992.0 0.281253 0.722820 10289 SH600306 2024-04-23 1022018.0 0.281253 0.686257 10290 SH600306 2024-04-24 1372334.0 0.281253 0.652507 10291 SH600306 2024-04-25 951008.0 0.281253 0.618756 10292 SH600306 2024-04-26 1968818.0 0.281253 0.587818 10293 SH600306 2024-04-29 1532764.0 0.281253 0.559693 10294 SH600306 2024-04-30 188390272.0 0.281253 0.098438 10295 SH600306 2024-05-06 117053368.0 0.281253 0.092813 10296 SH600306 2024-05-07 99965448.0 0.281253 0.101251 10297 SH600306 2024-05-08 85975896.0 0.281253 0.092813 10298 SH600306 2024-05-09 46003664.0 0.281253 0.095626 10299 SH600306 2024-05-10 61825620.0 0.281253 0.092813 10300 SH600306 2024-05-13 26138518.0 0.281253 0.092813 10301 SH600306 2024-05-14 19884768.0 0.281253 0.095626 10302 SH600306 2024-05-15 24197052.0 0.281253 0.090001 10303 SH600306 2024-05-16 12483558.0 0.281253 0.090001 10304 SH600306 2024-05-17 9390678.0 0.281253 0.087188 10305 SH600306 2024-05-20 27141916.0 0.281253 0.081563

I am hoping to debug dump_bin.py to find the problem. I ran it to here，the following code may be problem.

    def _data_to_bin(self, df: pd.DataFrame, calendar_list: List[pd.Timestamp], features_dir: Path):
        if df.empty:
            logger.warning(f"{features_dir.name} data is None or empty")
            return
        if not calendar_list:
            logger.warning("calendar_list is empty")
            return
        # align index
        _df = self.data_merge_calendar(df, calendar_list)
        if _df.empty:
            logger.warning(f"{features_dir.name} data is not in calendars")
            return

When align index, calendar_list does not contain dates such as 2024-05-06, but SH600306 data is empty in these days.

[SunsetWolf]

My guess is that your data is not normalized causing this issue. I tried using the command: python scripts/get_data.py qlib_data --target_dir <user data dir> --region cn Download the data, confirm that SH600306 exists in this data, and then use the command: python scripts/data_collector/yahoo/collector.py update_data_to_bin --qlib_data_1d_dir <user data dir> --end_date <end date> Performing an incremental update on the downloaded data did not happen as you described. It is recommended to use this method for incremental updates to the data.