pygetpapers
pygetpapers copied to clipboard
petermr
ResponseWrapper error (from EPMC)
THIS IS A GOOD EXAMPLE OF HOW TO REPORT A BUG
Describe the bug Query in pygetpapers in pygetpapers_literature_search.ipynb from Renu Kumari https://colab.research.google.com/drive/1-vM3BKV7NjvFXAdLGuqyNMh4VhPq6uMa?usp=sharing
To Reproduce Steps to reproduce the behavior:
- Go to https://colab.research.google.com/drive/1-vM3BKV7NjvFXAdLGuqyNMh4VhPq6uMa?usp=sharing
- Launch first 2 cells to install/help
- Run cell 3
!pygetpapers -n -q "carbon emission"
- Produces
Traceback (most recent call last):
File "/usr/local/bin/pygetpapers", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/pygetpapers/pygetpapers.py", line 537, in main
callpygetpapers.create_argparser()
File "/usr/local/lib/python3.10/dist-packages/pygetpapers/pygetpapers.py", line 530, in create_argparser
self.runs_pygetpapers_for_given_args(self.query_namespace)
File "/usr/local/lib/python3.10/dist-packages/pygetpapers/pygetpapers.py", line 312, in runs_pygetpapers_for_given_args
api_handler.check_query_logic_and_run()
File "/usr/local/lib/python3.10/dist-packages/pygetpapers/pygetpapers.py", line 183, in check_query_logic_and_run
self.api.noexecute(self.query_namespace)
File "/usr/local/lib/python3.10/dist-packages/pygetpapers/repository/europe_pmc.py", line 266, in noexecute
totalhits = result[RESPONSE_WRAPPER][HITCOUNT]
KeyError: 'responseWrapper'
Expected behavior I expected message that 10 hits had been found
Desktop (please complete the following information):
- OS: MacOS
- Browser Chrome
- Version 123.0.6312.59 (Official Build) (arm64)
Additional context This might be an error at EPMC - I will retry and also read their mailing list
Ran query immediately afterwards on EPMC. Got expected output.
Preprints (1,150)
Books & documents (1)
Date
2024 (924)
2023 (4,194)
2022 (3,963)
[China's wetland soil organic carbon pool: New estimation on pool size, change, and trajectory](https://europepmc.org/article/AGR/IND608163566)
[Ren Y](https://europepmc.org/search?query=AUTH%3A%22Ren%20Y%22), [Mao D](https://europepmc.org/search?query=AUTH%3A%22Mao%20D%22), [Wang Z](https://europepmc.org/search?query=AUTH%3A%22Wang%20Z%22), [Yu Z](https://europepmc.org/search?query=AUTH%3A%22Yu%20Z%22), [Xu X](https://europepmc.org/search?query=AUTH%3A%22Xu%20X%22), [Huang Y](https://europepmc.org/search?query=AUTH%3A%22Huang%20Y%22), [Xi Y](https://europepmc.org/search?query=AUTH%3A%22Xi%20Y%22), [Luo L](https://europepmc.org/search?query=AUTH%3A%22Luo%20L%22), [Jia M](https://europepmc.org/search?query=AUTH%3A%22Jia%20M%22), [Song K](https://europepmc.org/search?query=AUTH%3A%22Song%20K%22), [Li X](https://europepmc.org/search?query=AUTH%3A%22Li%20X%22)
Reran on local commandline without problems:
pm286macbook-2:~ pm286$ pygetpapers -q "carbon emissions" -k 10 -o junk -p
/opt/anaconda3/lib/python3.8/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (2.2.1) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
INFO: Total Hits are 84203
10it [00:00, 41859.32it/s]
0%| | 0/10 [00:00<?, ?it/s]INFO: Wrote the pdf file for PMC10907782
10%|████▍ | 1/10 [00:01<00:10, 1.19s/it]INFO: Wrote the pdf file for PMC10956835
20%|████████▊ | 2/10 [00:02<00:09, 1.17s/it]INFO: Wrote the pdf file for PMC10928123
INFO: Wrote the pdf file for PMC10928123
30%|█████████████▏ | 3/10 [00:26<01:20, 11.45s/it]INFO: Wrote the pdf file for PMC10925036
40%|█████████████████▌ | 4/10 [00:28<00:46, 7.77s/it]INFO: Wrote the pdf file for PMC10867074
INFO: Wrote the pdf file for PMC10867074
50%|██████████████████████ | 5/10 [00:32<00:32, 6.52s/it]INFO: Wrote the pdf file for PMC10655991
60%|██████████████████████████▍ | 6/10 [00:33<00:18, 4.74s/it]INFO: Wrote the pdf file for PMC10907649
70%|██████████████████████████████▊ | 7/10 [00:34<00:10, 3.57s/it]INFO: Wrote the pdf file for PMC10623158
80%|███████████████████████████████████▏ | 8/10 [00:38<00:06, 3.49s/it]INFO: Wrote the pdf file for PMC10883542
90%|███████████████████████████████████████▌ | 9/10 [00:39<00:02, 2.94s/it]INFO: Wrote the pdf file for PMC10902235
100%|███████████████████████████████████████████| 10/10 [00:41<00:00, 4.12s/it]
(base) pm286macbook-2:~ pm286$ tree junk
junk
├── PMC10623158
│ ├── eupmc_result.json
│ └── fulltext.pdf
├── PMC10655991
│ ├── eupmc_result.json
│ └── fulltext.pdf
├── PMC10867074
│ ├── eupmc_result.json
│ └── fulltext.pdf
├── PMC10883542
│ ├── eupmc_result.json
│ └── fulltext.pdf
├── PMC10902235
│ ├── eupmc_result.json
│ └── fulltext.pdf
├── PMC10907649
│ ├── eupmc_result.json
│ └── fulltext.pdf
├── PMC10907782
│ ├── eupmc_result.json
│ └── fulltext.pdf
├── PMC10925036
│ ├── eupmc_result.json
│ └── fulltext.pdf
├── PMC10928123
│ ├── eupmc_result.json
│ └── fulltext.pdf
├── PMC10956835
│ ├── eupmc_result.json
│ └── fulltext.pdf
└── eupmc_results.json
Reran the Colab notebook which worked satisfactorily , so this may be a transient problem.
### **Step 4: Downloading pdf only for 10 papers on the query.**
* **-k** for desired number of papers to work on
* **-q** for query term
* **-o** for output dir
* **-p** for pdf download
!pygetpapers -q "carbon emission" -k 10 -o "carbon_emi" -p
pm286macbook-2:~ pm286$ pygetpapers -q "carbon emissions" -k 10 -o junk -p /opt/anaconda3/lib/python3.8/site-packages/requests/init.py:87: RequestsDependencyWarning: urllib3 (2.2.1) or chardet (5.2.0) doesn't match a supported version! warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported " INFO: Total Hits are 84203 10it [00:00, 41859.32it/s] 0%| | 0/10 [00:00<?, ?it/s]INFO: Wrote the pdf file for PMC10907782 10%|████▍ | 1/10 [00:01<00:10, 1.19s/it]INFO: Wrote the pdf file for PMC10956835 20%|████████▊ | 2/10 [00:02<00:09, 1.17s/it]INFO: Wrote the pdf file for PMC10928123 INFO: Wrote the pdf file for PMC10928123 30%|█████████████▏ | 3/10 [00:26<01:20, 11.45s/it]INFO: Wrote the pdf file for PMC10925036 40%|█████████████████▌ | 4/10 [00:28<00:46, 7.77s/it]INFO: Wrote the pdf file for PMC10867074 INFO: Wrote the pdf file for PMC10867074 50%|██████████████████████ | 5/10 [00:32<00:32, 6.52s/it]INFO: Wrote the pdf file for PMC10655991 60%|██████████████████████████▍ | 6/10 [00:33<00:18, 4.74s/it]INFO: Wrote the pdf file for PMC10907649 70%|██████████████████████████████▊ | 7/10 [00:34<00:10, 3.57s/it]INFO: Wrote the pdf file for PMC10623158 80%|███████████████████████████████████▏ | 8/10 [00:38<00:06, 3.49s/it]INFO: Wrote the pdf file for PMC10883542 90%|███████████████████████████████████████▌ | 9/10 [00:39<00:02, 2.94s/it]INFO: Wrote the pdf file for PMC10902235 100%|███████████████████████████████████████████| 10/10 [00:41<00:00, 4.12s/it] (base) pm286macbook-2:~ pm286$ tree junk junk ├── PMC10623158 │ ├── eupmc_result.json │ └── fulltext.pdf ├── PMC10655991 │ ├── eupmc_result.json │ └── fulltext.pdf ├── PMC10867074 │ ├── eupmc_result.json │ └── fulltext.pdf ├── PMC10883542 │ ├── eupmc_result.json │ └── fulltext.pdf ├── PMC10902235 │ ├── eupmc_result.json │ └── fulltext.pdf ├── PMC10907649 │ ├── eupmc_result.json │ └── fulltext.pdf ├── PMC10907782 │ ├── eupmc_result.json │ └── fulltext.pdf ├── PMC10925036 │ ├── eupmc_result.json │ └── fulltext.pdf ├── PMC10928123 │ ├── eupmc_result.json │ └── fulltext.pdf ├── PMC10956835 │ ├── eupmc_result.json │ └── fulltext.pdf └── eupmc_results.json
This is generated because RESPONSE_WRAPPER is missing/None. If get(RESPONSE_WRAPPER) is used it would return None and this could be tested rather than trapping the exception.
I hope that this is a transient error generated from EPMC.