turicreate
turicreate copied to clipboard
apple
better progress report
update 05/08/2020
implemented a thread-safe stopwatch to measure the elapsed time spent by concurrent IO.
report progress to users in order to let them wait for slow IO operations. Reduce the chance that users think the program is hanging without progress.
Tries to solve #3119. But due to the fact that we should provide prompt based on elapsed time. Anyway, we should have metrics inside of the SFrame by remembering how many bytes are downloaded and tell the user the progress.
update
upload
05/08/2020
Finished fetching block 0. Elapsed 0s for downloading s3://tc_qa/integration/manual/upload/medium_sframe_ac/dir_archive.ini
Uploading s3://tc_qa/integration/manual/upload/medium_sframe_ac/dir_archive.ini. Elapsed time 0 seconds
Finished uploading s3://tc_qa/integration/manual/upload/medium_sframe_ac/dir_archive.ini. Elapsed time 0 seconds
Uploading s3://tc_qa/integration/manual/upload/medium_sframe_ac/m_9cfcba377b62f99.0000. Elapsed time 0 seconds
Finished uploading s3://tc_qa/integration/manual/upload/medium_sframe_ac/m_9cfcba377b62f99.0000. Elapsed time 69 seconds
Uploading s3://tc_qa/integration/manual/upload/medium_sframe_ac/m_9cfcba377b62f99.sidx. Elapsed time 0 seconds
Finished uploading s3://tc_qa/integration/manual/upload/medium_sframe_ac/m_9cfcba377b62f99.sidx. Elapsed time 0 seconds
Uploading s3://tc_qa/integration/manual/upload/medium_sframe_ac/m_9cfcba377b62f99.frame_idx. Elapsed time 0 seconds
Finished uploading s3://tc_qa/integration/manual/upload/medium_sframe_ac/m_9cfcba377b62f99.frame_idx. Elapsed time 0 seconds
Uploading s3://tc_qa/integration/manual/upload/medium_sframe_ac/dir_archive.ini. Elapsed time 0 seconds
Finished uploading s3://tc_qa/integration/manual/upload/medium_sframe_ac/dir_archive.ini. Elapsed time 0 seconds
download
a prompt based on time interval.
Finished fetching block 0. Elapsed 0s for downloading s3://tc_qa/integration/manual/sframes/big_sframe_od/dir_archive.ini
Finished fetching block 0. Elapsed 0s for downloading s3://tc_qa/integration/manual/sframes/big_sframe_od/m_ea214100aab1ac60.sidx
Finished fetching block 0. Elapsed 0s for downloading s3://tc_qa/integration/manual/sframes/big_sframe_od/m_ea214100aab1ac60.frame_idx
Finished fetching block 1. Elapsed 10s for downloading s3://tc_qa/integration/manual/sframes/big_sframe_od/m_ea214100aab1ac60.0000
Finished fetching block 0. Elapsed 74s for downloading s3://tc_qa/integration/manual/sframes/big_sframe_od/m_ea214100aab1ac60.0000
+------------------------+--------------+----------+
| image | name | label |
+------------------------+--------------+----------+
| Height: 480 Width: 640 | bike_248 | bike |
| Height: 480 Width: 640 | bike_254 | bike |
| Height: 480 Width: 640 | bike_267 | bike |
| Height: 480 Width: 640 | bike_107 | bike |
| Height: 480 Width: 640 | carsgraz_225 | carsgraz |
| Height: 480 Width: 640 | bike_187 | bike |
| Height: 480 Width: 640 | bike_077 | bike |
| Height: 480 Width: 640 | carsgraz_053 | carsgraz |
| Height: 480 Width: 640 | bike_362 | bike |
| Height: 480 Width: 640 | carsgraz_351 | carsgraz |
+------------------------+--------------+----------+
@guihao-liang - As we discussed it would in person, it would be better if progress was reported at fixed time interval rather then every time a 64MB block is uploaded/downloaded. Here is an example of where we do something like that in the Python layer.
@guihao-liang - As we discussed it would in person, it would be better if progress was reported at fixed time interval rather then every time a 64MB block is uploaded/downloaded. Here is an example of where we do something like that in the Python layer.
Thanks! I already had my own implementation yesterday. But it's pretty similar. I think a better solution is to have a separate thread reporting every 20s. But I have no time. This should be my last S3 PR. I need to work on other stuff.