Retrieve Transportation Costs

[trevorb1]

Checklist

• [X] I am using the current master branch
• [X] I am running on an up-to-date pypsa-usa environment. Update via conda env update -f envs/environment.yaml

The Issue

The download of transportation costs is sometimes throwing retrieval errors. As a temporary fix for issue #95, the transportation costs were commented out since they are not currently used. Once transportation is added to the system, this issue will need to be addressed.

Note, that this behaviour was somewhat erratic. Sometimes it worked, and sometimes it didnt when I have tested over the last couple hours.

Steps To Reproduce

1. Uncomment the retrieval of transportation costs in the rule retrieve_costs_usa and accompanying script
2. Run the workflow

Expected Behavior

I would expect the xlsx file to download to resources/ without error

Error Message

Traceback (most recent call last):
  File "/Users/nomio/Documents/REU/Fall_2324/pypsa-usa/workflow/.snakemake/scripts/tmpt25gjbej.retrieve_cost_data_usa.py", line 133, in <module>
    progress_retrieve(xlsx, save_atb_transport)
  File "/Users/nomio/Documents/REU/Fall_2324/pypsa-usa/workflow/scripts/_helpers.py", line 223, in progress_retrieve
    urllib.request.urlretrieve(url, file, reporthook=dlProgress)
  File "/Users/nomio/miniforge3/envs/pypsa-usa/lib/python3.11/urllib/request.py", line 241, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
                            ^^^^^^^^^^^^^^^^^^
  File "/Users/nomio/miniforge3/envs/pypsa-usa/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nomio/miniforge3/envs/pypsa-usa/lib/python3.11/urllib/request.py", line 519, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nomio/miniforge3/envs/pypsa-usa/lib/python3.11/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nomio/miniforge3/envs/pypsa-usa/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/Users/nomio/miniforge3/envs/pypsa-usa/lib/python3.11/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nomio/miniforge3/envs/pypsa-usa/lib/python3.11/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
100% (100 of 100) |#####################################################| Elapsed Time: 0:00:00 ETA:  00:00:00
[Thu Oct 12 15:32:37 2023]
Error in rule retrieve_cost_data_usa:
    jobid: 17
    output: resources/costs/nrel_atb.parquet, resources/costs/nrel_atb_transport.xlsx, resources/costs/ng_electric_power_price.csv, resources/costs/ng_industrial_price.csv, resources/costs/ng_commercial_price.csv, resources/costs/ng_residential_price.csv
    log: logs/retrieve_cost_data_usa.log (check log file(s) for error details)

RuleException:
CalledProcessError in file /Users/nomio/Documents/REU/Fall_2324/pypsa-usa/workflow/Snakefile, line 218:
Command 'set -euo pipefail;  /Users/nomio/miniforge3/envs/pypsa-usa/bin/python3.11 /Users/nomio/Documents/REU/Fall_2324/pypsa-usa/workflow/.snakemake/scripts/tmpt25gjbej.retrieve_cost_data_usa.py' returned non-zero exit status 1.
  File "/Users/nomio/Documents/REU/Fall_2324/pypsa-usa/workflow/Snakefile", line 218, in __rule_retrieve_cost_data_usa
  File "/Users/nomio/miniforge3/envs/pypsa-usa/lib/python3.11/concurrent/futures/thread.py", line 58, in run
Removing output files of failed job retrieve_cost_data_usa since they might be corrupted:
resources/costs/nrel_atb.parquet
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-10-12T153223.004632.snakemake.log

----------------------------------------------------------------------------------------------------

<urlopen error [Errno -3] Temporary failure in name resolution>
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

  File "/home/trevor/repos/pypsa-usa/workflow/scripts/_helpers.py", line 223, in progress_retrieve
    urllib.request.urlretrieve(url, file, reporthook=dlProgress)
  File "/home/trevor/repos/pypsa-usa/workflow/scripts/retrieve_cost_data_usa.py", line 129, in <module>
    progress_retrieve(xlsx, save_atb_transport)
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>

Anything else?

Since the data for the transportation costs is not hosted on AWS like the rest of the NREL data, rather it is just a download link from a website, it may be better to move this to the sector coupled Zenodo deposit.