MarcYin
HTTPError when downloading VIIRS data
Hi @MarcYin @jscarriere
Could you handle the error below so that the pipeline doesn't fail as a result? 9acc5883-1d65-4d5f-9e55-7964c3d0fde0: S2B_MSIL1C_20250305T100029_N0511_R122_T33UWP_20250305T115712
FYI: I relaunched the pipeline and it's done
{"asctime": "2025-04-09 16:25:03,099", "levelname": "DEBUG", "filename": "connectionpool.py", "funcName": "make_request", "message": "https://d1nklfio7vscoe.cloudfront.net:443 "GET /s3-2d2df3a34830d5223d1e9547cd713408/lp-prod-protected.s3.us-west-2.amazonaws.com/VNP43MA1.002/VNP43MA1.A2025080.h19v04.002.2025088094537/VNP43MA1.A2025080.h19v04.002.2025088094537.h5?A-userid=s2gmservice&Expires=1744219502&Signature=rHK2S7EJGUA9rMNGrkMYFPuKMqulUEzs~VcarWhlspsOpctQC392uS19yexSu0Jctb-62LRwfjDnGp7iGaIg52THqd1BdxZI44tATV17rpq9TihQbAlkf5g6KpLDoxsvag4IEsHQROz3C1TYvCE2HoaA4Nxm4dTiAVl5aRFVKIA1QqJI~tWjIkrtd6MxUeNg0DJWPA69gscb1BpmShM72mCxiWcCZDtIdpfDkuNZWXno5~EOi84rYvbDqNRFomFXJvODcWmRagpG67RQ0MHcMaDuG0x9iQ2ue2ci7q7JIORvi7N7JqIy6MaxMJAxBVxjHV~zSlaJzWecnL5jYvcsIw_&Key-Pair-Id=K1IY6VK8YG6XAS HTTP/1.1" 200 57327104", "pipeline": "siac_pipeline"}
{"asctime": "2025-04-09 16:25:05,977", "levelname": "ERROR", "filename": "siac_pipeline.py", "funcName": "siac_pipeline", "message": "An exception occured during the execution of the pipeline", "pipeline": "siac_pipeline", "exc_info": "Traceback (most recent call last):\n File "/usr/local/lib/python3.10/dist-packages/earthaccess/store.py", line 697, in download_file\n r.raise_for_status()\n File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status\n raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://d1nklfio7vscoe.cloudfront.net/s3-2d2df3a34830d5223d1e9547cd713408/lp-prod-protected.s3.us-west-2.amazonaws.com/VNP43MA1.002/VNP43MA1.A2025065.h18v04.002.2025073095903/VNP43MA1.A2025065.h18v04.002.2025073095903.h5?A-userid=s2gmservice&Expires=1744219487&Signature=3~zyWvgAtI1r4BPcZa0GzWN3VrZqp1ANph76oapiUh2x1ZQ~-VyPdvRqChT2pP4xNmfrLszqGaxys1L1OL4V6--M6APw0QDEdb7SCVife~dBnTlFDFhey3loDvWpSJc-0CBl4jDZGbb7tgB7daBmeQByWROvftnOrX3OfNoNSs0zK3i61O7dRXjAgLWAqIm5if7ulHomKQS0W6c-uyILSqzpqTcMa4h0vudH~YQdJcNk8g591lFNfpXXGhCaQO7dBMD2ci6FN3D~LupFKZKbRT2AbFVJQ4UZLF-FirgccbGA66wJgZu9kMgFaT58riBles7E-~USJl2qGxqw1qaHKw_&Key-Pair-Id=K1IY6VK8YG6XAS\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/opt/s2gm/siac/src/siac_pipeline.py", line 162, in siac_pipeline\n SIAC_S2(\n File "/opt/s2gm/modules/SIAC/SIAC/SIAC_S2.py", line 49, in SIAC_S2\n aero_atmo = do_correction(*ret)\n File "/opt/s2gm/modules/SIAC/SIAC/SIAC_S2.py", line 187, in do_correction\n filenames = download_VNP43MA1(toa_refs[0], mcd43_date, mcd43, temporal_window = 16)\n File "/opt/s2gm/modules/SIAC/SIAC/get_VNP43MA1.py", line 498, in download_VNP43MA1\n filenames = earthaccess.download(to_download, VNP43_dir)\n File "/usr/local/lib/python3.10/dist-packages/earthaccess/api.py", line 270, in download\n return earthaccess.store.get(\n File "/usr/local/lib/python3.10/dist-packages/earthaccess/store.py", line 535, in get\n return self._get(granules, Path(local_path), provider, pqdm_kwargs=pqdm_kwargs)\n File "/usr/local/lib/python3.10/dist-packages/multimethod/init.py", line 350, in call\n return func(*args, **kwargs)\n File "/usr/local/lib/python3.10/dist-packages/earthaccess/store.py", line 649, in _get_granules\n return self._download_onprem_granules(\n File "/usr/local/lib/python3.10/dist-packages/earthaccess/store.py", line 747, in _download_onprem_granules\n return pqdm(arguments, self._download_file, **pqdm_kwargs)\n File "/usr/local/lib/python3.10/dist-packages/pqdm/threads.py", line 22, in pqdm\n return _parallel_process(\n File "/usr/local/lib/python3.10/dist-packages/pqdm/_base.py", line 79, in _parallel_process\n raise e\n File "/usr/local/lib/python3.10/dist-packages/pqdm/_base.py", line 76, in _parallel_process\n results.append(future.result())\n File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result\n return self.__get_result()\n File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result\n raise self._exception\n File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/usr/local/lib/python3.10/dist-packages/earthaccess/store.py", line 705, in _download_file\n raise Exception\nException"}
