Job outputs should be stored under a single Job Directory

[aronchick]

I've been having a weird bug around "get".

I ran this job:

    URL=https://picsum.photos/500/500/

    # Add the image to bacalhau
    bacalhau docker run \
    --id-only
    --input-urls $URL \
    -l $LABEL \
    ubuntu -- cp -rv /inputs/. /outputs/

Which resulted in this output:

Job successfully submitted. Job ID: 19d487d6-e4b7-4a28-a802-1c1691005f66
Checking job status... (Enter Ctrl+C to exit at any time, your job will continue running):

               Creating job for submission ... done ✅
               Finding node(s) for the job ... done ✅
                     Node accepted the job ... done ✅
                                           ... done ✅
           Job finished, verifying results ... done ✅
              Results accepted, publishing ... done ✅
                                          
Results CID: Qmd4SK6ayX2aYD1RDfstsoJnqH8ifxvCVPytcxy87p2a1p
Job Results By Node:
Node QmXaXu9N:
  Shard 0:
    Status: Cancelled
    No RunOutput for this shard
Node QmYgxZiy:
  Shard 0:
    Status: Completed
    Container Exit Code: 0
    Stdout:
      '/inputs/./500' -> '/outputs/./500'
    Stderr: <NONE>
Node QmdZQ7Zb:
  Shard 0:
    Status: Cancelled
    No RunOutput for this shard

To download the results, execute:
  bacalhau get 19d487d6-e4b7-4a28-a802-1c1691005f66

To get more details about the run, execute:
  bacalhau describe 19d487d6-e4b7-4a28-a802-1c1691005f66

But when I get it:

❯ bacalhau get 5aa64ede-4844-492d-a4c2-dba71be08cae
Fetching results of job '5aa64ede-4844-492d-a4c2-dba71be08cae'...%    

I get nothing in the directory:

❯ ls -lR job-5aa64ede-4844-492d-a4c2-dba71be08cae-shard-0-host-QmYgxZiySj3MRkwLSL4X2MF5F9f2PMhAE3LV49XkfNL1o3
drwxr-xr-x - daaronch 12 Oct 16:43 outputs

job-5aa64ede-4844-492d-a4c2-dba71be08cae-shard-0-host-QmYgxZiySj3MRkwLSL4X2MF5F9f2PMhAE3LV49XkfNL1o3/outputs:
  ~/code/bacalhau-examples/stablediffusiongallery   stablediffusiongallery !1 ?1                                       
❯ 

BUT, it looks like it's there:

https://gateway.ipfs.io/ipfs/Qmd4SK6ayX2aYD1RDfstsoJnqH8ifxvCVPytcxy87p2a1p

---

Ok, this was my fault - I thought everything would be listed under a single job directory, so i missed the "shard" and "volume" directories. I think we're storing things inverted - it should be:

• Job ID
  • Stuff
• Job ID 2
  • Stuff

Not

• Volume
  • Job ID
  • Job ID 2

Etc.

[simonwo]

How does this interact with #585?

[lukemarsden]

this is the same as https://github.com/filecoin-project/bacalhau/issues/585