metaflow Tutorial episode 6 does not work in AWS because of MetaflowS3URLException

I was able to successfully run episode 5 with the Sandbox environment provided by the metaflow team. However, when I run episode 6, the "start" step fails with the following exception: metaflow.datatools.s3.MetaflowS3URLException: Initialize S3 with an 's3root' or 'run' if you don't want to specify full s3:// urls. The error is raised in this source code.

I have tried this from 2 different laptops, always closely following the instructions from the episode 6 instructions. In both cases, I have received the same error. So, I am wondering if this might be related to the Sandbox environment that was provided to me.

Stacktrace:

python 02-statistics/stats.py --environment conda run --with batch --max-workers 4 --with conda:python=3.7,libraries="{pandas:0.24.2}"
Metaflow 2.7.7 executing MovieStatsFlow for user:myusername
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2022-08-25 19:02:12.091 Creating local datastore in current directory (/home/myusername/Desktop/metaflow-tutorials/.metaflow)
Bootstrapping conda environment...(this could take a few minutes)
Including file /home/myusername/Desktop/metaflow-tutorials/02-statistics/movies.csv of size 191KB 
File persisted at s3://sc-882357307949-pp-nk4vp7nc6535e-metaflows3bucket-1sc74r769qij3/metaflow/data/MovieStatsFlow/9e3a38eb45be19331cf19b394d8191f54a06a281
2022-08-25 19:04:15.127 Workflow starting (run-id 8):
2022-08-25 19:04:18.547 [8/start/52 (pid 11234)] Task is starting.
2022-08-25 19:04:21.001 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1] Task is starting (status SUBMITTED)...
2022-08-25 19:04:21.964 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1] Task is starting (status RUNNABLE)...
2022-08-25 19:04:25.484 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1] Task is starting (status STARTING)...
2022-08-25 19:04:55.496 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1] Task is starting (status STARTING)...
2022-08-25 19:05:05.282 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1] Task is starting (status RUNNING)...
2022-08-25 19:05:03.646 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1] Setting up task environment.
2022-08-25 19:05:44.815 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1] Downloading code package...
2022-08-25 19:05:45.597 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1] Code package downloaded.
2022-08-25 19:05:45.621 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1] Task is starting.
2022-08-25 19:05:46.272 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1] Bootstrapping environment...
2022-08-25 19:05:46.313 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1] Traceback (most recent call last):
2022-08-25 19:05:49.619 [8/start/52 (pid 11234)] AWS Batch error:
2022-08-25 19:05:49.803 [8/start/52 (pid 11234)] Essential container in task exited This could be a transient error. Use @retry to retry.
2022-08-25 19:05:49.803 [8/start/52 (pid 11234)] 
2022-08-25 19:05:46.313 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]   File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
2022-08-25 19:05:46.314 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]     return _run_code(code, main_globals, None,
2022-08-25 19:05:46.314 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]   File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
2022-08-25 19:05:46.314 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]     exec(code, run_globals)
2022-08-25 19:05:46.314 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]   File "/metaflow/metaflow/plugins/conda/batch_bootstrap.py", line 91, in <module>
2022-08-25 19:05:46.314 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]     bootstrap_environment(sys.argv[1], sys.argv[2], sys.argv[3])
2022-08-25 19:05:46.314 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]   File "/metaflow/metaflow/plugins/conda/batch_bootstrap.py", line 16, in bootstrap_environment
2022-08-25 19:05:46.314 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]     packages = download_conda_packages(flow_name, env_id, datastore_type)
2022-08-25 19:05:46.314 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]   File "/metaflow/metaflow/plugins/conda/batch_bootstrap.py", line 46, in download_conda_packages
2022-08-25 19:05:46.314 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]     for pkg in s3.get_many(env["cache_urls"]):
2022-08-25 19:05:46.314 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]   File "/metaflow/metaflow/datatools/s3.py", line 935, in get_many
2022-08-25 19:05:46.314 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]     return list(starmap(S3Object, _get()))
2022-08-25 19:05:46.314 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]   File "/metaflow/metaflow/datatools/s3.py", line 902, in _get
2022-08-25 19:05:46.315 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]     for s3prefix, s3url, fname in res:
2022-08-25 19:05:46.315 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]   File "/metaflow/metaflow/datatools/s3.py", line 1264, in _read_many_files
2022-08-25 19:05:46.315 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]     prefixes_and_ranges = list(prefixes_and_ranges)
2022-08-25 19:05:46.315 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]   File "/metaflow/metaflow/datatools/s3.py", line 554, in _url_and_range
2022-08-25 19:05:46.315 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]     url = self._url(key_value)
2022-08-25 19:05:46.315 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]   File "/metaflow/metaflow/datatools/s3.py", line 539, in _url
2022-08-25 19:05:46.316 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1]     raise MetaflowS3URLException(
2022-08-25 19:05:46.316 [8/start/52 (pid 11234)] [3547076b-42e2-4b67-955b-c27c071611b1] metaflow.datatools.s3.MetaflowS3URLException: Initialize S3 with an 's3root' or 'run' if you don't want to specify full s3:// urls.
2022-08-25 19:05:50.516 [8/start/52 (pid 11234)] Task failed.
2022-08-25 19:05:51.191 Workflow failed.
2022-08-25 19:05:51.191 Terminating 0 active tasks...
2022-08-25 19:05:51.192 Flushing logs...
    Step failure:
    Step start (task-id 52) failed.

Aug 25 '22 17:08 juliuskittler

#1128 will fix this issue.

Sep 03 '22 00:09 savingoyal

fix seems to have been merged. feel free to reopen if this is still an issue

Aug 30 '23 09:08 saikonen