Finish creating project before downloading image.

[xerrni]

Currently when project is created a POST request is sent. Response for this request is sent after input is downloaded.

In my opinion it is wrong user experience as download can take long time (huge docker images). I propose that web container should return page with created project before image is downloaded.

Current behavior breaks some nginx configuration as timeout occurs in nginx before it receives response from web container and it returns 504 error code (gateway timeout) as response.

Furthermore when user hits refresh button request is resend and web re-downloads this input.

[pombredanne]

@xerrni

In my opinion it is wrong user experience as download can take long time (huge docker images). I propose that web container should return page with created project before image is downloaded.

I completely agree. Large images (including huge VM or Docker images that can routinely be 30 to 50GB big) are problematic. There are a few behaviours possible:

1. download in the background after the project has been created
2. kick in only when the pipeline is executed, download as a pre-pipeline process if there are downloadable URLs
3. kick in only when the pipeline is executed, download as a first explicit step of a pipeline if there are downloadable URLs. A pipeline without a download step would not download.

Which ones would be the better way?

@cco3 what's your take there?

[cco3]

I'm not sure I get the distinctions in the options, but returning immediately with a project created seems to make the most sense...the download could just be the first stage in the pipeline.

[aalexanderr]

I think 3rd option is best - fetching is already a pipe and it would make scaling amount of workers easier based on amount of runs to process

[pombredanne]

See also https://github.com/nexB/scancode.io/issues/372

[tdruez]

The download of the Project's inputs now takes place in the pipeline run.