stock
stock copied to clipboard
myhhub
修复任务告警信息；调整数据库配置读取方式以降低配置项对py代码的侵入。
昨天闲逛公众号了解到本项目，今天clone下来跑了一下，很厉害很不错。
修复警告
在使用过程中发现了2处任务执行警告对其进行修复，警告如下：
------正在执行作业中，请等待------
0%| | 0/6 [00:00<?, ?it/s]\instock\core\crawling\stock_lhb_sina.py:103: FutureWarning: Passing literal html to 'read_html' is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
temp_df = pd.read_html(r.text)[0].iloc[0:, :]
\instock\core\crawling\stock_lhb_sina.py:103: FutureWarning: Passing literal html to 'read_html' is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
temp_df = pd.read_html(r.text)[0].iloc[0:, :]
33%|██████████████████████████████████████ | 2/6 [00:00<00:00, 11.27it/s]\instock\core\crawling\stock_lhb_sina.py:103: FutureWarning: Passing literal html to 'read_html' is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
temp_df = pd.read_html(r.text)[0].iloc[0:, :]
\instock\core\crawling\stock_lhb_sina.py:103: FutureWarning: Passing literal html to 'read_html' is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
temp_df = pd.read_html(r.text)[0].iloc[0:, :]
67%|████████████████████████████████████████████████████████████████████████████ | 4/6 [00:00<00:00, 12.99it/s]\instock\core\crawling\stock_lhb_sina.py:103: FutureWarning: Passing literal html to 'read_html' is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
temp_df = pd.read_html(r.text)[0].iloc[0:, :]
\instock\core\crawling\stock_lhb_sina.py:103: FutureWarning: Passing literal html to 'read_html' is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
temp_df = pd.read_html(r.text)[0].iloc[0:, :]
50%|█████████████████████████████████████████████████████████ | 1/2 [00:00<00:00, 2.30it/s]\instock\core\crawling\stock_fhps_em.py:49: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
big_df = pd.concat([big_df, temp_df], ignore_index=True)
\instock\core\crawling\stock_fhps_em.py:49: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
big_df = pd.concat([big_df, temp_df], ignore_index=True)
调整数据库配置
另外对database.py的配置逻辑做了微调，理由是： 脚本中存在数据库配置，若配置需要修改或调整就需要调整py代码。一方面，调整代码仅用于修改数据库配置的话，应该额外放置到配置文件中。另一方面，如果作者或其他贡献者因种种原因修改了配置项，然后在提交代码时忘记隐去相关配置则存在隐私信息泄露安全隐患。
因此，做了如下微调：
在database.py同目录下创建默认配置模板文件database_default.json，该文件为数据库连接信息配置。使用时将其重命名为database.json供database.py读取使用。通过.gitignore增加对database.json的过滤来实现隐私信息的本地化。
而在database.py中若用户并未使用database.json则采用默认值的方式向前兼容，确保现有用户（未自定义数据库信息）的正常使用。
