Implement imageUri on mint process

[andrevenancio]

Implementing imageUri like we did with the HTML version will enable us to achieve a couple of things:

Speed up how fast assets load on the main feed and on artist profile. It will also allow us to add a cover image to video formats, and further optimise the site speed by displaying the cover image only if the video is in view.

[tarwin]

For images this could be auto-generated in HTML on upload in JS. For videos it is more complicated, but could require that an image is also uploaded along with video.

[veqtor]

For images this could be auto-generated in HTML on upload in JS. For videos it is more complicated, but could require that an image is also uploaded along with video.

Would be nice to automatically generate a DASH manifest for videos, or just simply a preview video at a fixed, lower bitrate.

[tarwin]

I'm looking at using FFMPEG.wasm to generate a preview video of a lower bitrate, fixed length and max size. And do the same for images.

[andrevenancio]

For images this could be auto-generated in HTML on upload in JS. For videos it is more complicated, but could require that an image is also uploaded along with video.

I'm tempted to request it regardless of the media. I dont want to end up having multiple forms for multiple media types. Having the thumbnail field like we have the title, description, tags would (in my opinion) be better.

[andrevenancio]

But that lower bitrate video features sound good! Does it mean that we need to convert it at mint time, and upload 2 assets?

[tarwin]

But that lower bitrate video features sound good! Does it mean that we need to convert it at mint time, and upload 2 assets?

We should be able to create a new video in the browser before they even mint it using FFMPEG.wasm

[pichiste]

Re: how this all gets saved in the metadata if we're trying to follow the spec:

Currently the displayUri field is used for HTML cover images, so I guess that should be the same across other media types.

The thumbnailUri field is what is displayed in wallets and is meant to be thumbnail size. Maybe we can auto-generate this one too? Currently it's just a placeholder image.

Where would the low-bitrate video thumbnail reference be stored? Seems like externalUri makes the most sense but not sure.

[andrevenancio]

can we store it all in the same place? The wallets check a certain property and that's what they use to show the preview of the asset. I think its displayUri. Why not use the same for all media? Also, based on my experience I would NOT automate the process to create screenshots for videos. I would get the user to supply the frame they like. Otherwise we then need to implement a way of them changing the generated video preview if they dont like it. making the displayUri mandatory on all media allows us to have a preview on the wallets, and have a faster loading feed where we just display the displayUri and then on the objkt detail page we load the full asset.

[pichiste]

I think wallets grab thumbnailUri, at least Kukai does (it's the little black dot):

https://ipfs.io/ipfs/QmNrhZHUaEqxhyLfqoq1mtHSipkWHeT31LNHb1QEbDHgnc/

All I meant to say is that the thumbnailUri should just be an (auto-generated) mini version of displayUri, which is what the user uploads manually. Just two sizes of same image, each with a different purpose.

Re: preview videos, technically the spec expects image assets for both thumbnailUri and displayUri, so that's why I'm wondering what's the right place for a proper video? Maybe displayUri is ok?

[pichiste]

Hm, using displayUri might end up breaking something on a wallet if they are expecting an image actually.

Maybe the preview video ref should just be added to attributes? https://tzip.tezosagora.org/proposal/tzip-21/#attributes-array

[tarwin]

thumbnailUri: we could always create small animated GIFs or even better WebP from videos. Choose 10 frames from throughout the video and recompress.

[pichiste]

@tarwin how far did you get with looking into ffmpeg.wasm? Seems like the best comprehensive solution for displayUri / thumbnailUri / video thumbnails.

There is now some basic cover/thumb image generation and the automated compression part could easily be swapped out for this.

Only issue is browser support, since it depends on SharedBufferArray and there are known issues on older versions of Firefox. One solution could be to fall back to the current "Upload Cover Image" for those cases (Safari, IE, old FF).

[tarwin]

@pichiste I haven't had a lot of time to look. Been working with the backend team to get that stuff faster etc.

I'd suggest we require displayUri for everything. If it isn't supported yet ie videos let people upload something. Restrict it to less than 1mb or some such. Could be a video, an image, a GIF etc.

We can then replace with auto-created at some point.

You are right about browser support. That said, I don't think you should worry too much about browser support. A lot of stuff will break there including certain video encodings, some SVG stuff etc. Also, given the massive amounts of data being downloaded currently I don't see the site supporting anyone without a good internet connection (as it is at least) and they will have a greenfield browser.

At least for uploading. For older browsers we could just make them upload displayUri.

[pichiste]

@tarwin Got it, sounds good. The cover/thumb feature is disabled on live version, but displayUri is already required across media types via a manual upload on the mint page (except for images), so it's really just about getting video thumbnails working and automating as much as possible.

I can take a look at this when I find some time. Happy to collaborate on this as well, I'm on the discord.

[pichiste]

📝 Updated 2020-04-23 with feedback from @tarwin and @veqtor

Here is a rough spec proposal based on conversations here and on discord:

Engine

Existing (but disabled) cover/thumbnail image generation util switches to using ffmpeg.wasm. The consequence is minting won't work on some browsers, but we can add messaging for this and guide people to use a supported browsers.

Optimized Images

All media types generate a set of optimized images at these sizes:

max 256w
max 512w
max 1024w
max 2048w

These should be compressed jpegs, except in the case of animated GIFs. Images won't be generated in cases that would require upscaling.

The IPFS hashes of these images get stored in the OBJKT metadata under formats, while the hashes for 256 and 1024 additionally get stored in thumbnailUri and displayUri respectively:

{
  "artifactUri": "ipfs://hash-of-original",
  "displayUri": "ipfs://hash-of-1024",
  "thumbnailUri": "ipfs://hash-of-256",
  ...
  "formats": [
    {
      "mimeType": "image/jpeg",
      "fileName": "sm",
      "uri": "ipfs://hash-of-256",
      "hash": "hash-of-256",
      "fileSize": "xxxxxxxxxx",
      "dimensions": {
        "value": "256x192",
        "unit": "px"
      }
    },
    {
      "mimeType": "image/jpeg",
      "fileName": "md",
      "uri": "ipfs://hash-of-512",
      "hash": "hash-of-512",
      "fileSize": "xxxxxxxxxx",
      "dimensions": {
        "value": "512x384",
        "unit": "px"
      }      
    },
    {
      "mimeType": "image/jpeg",
      "fileName": "lg",
      "uri": "ipfs://hash-of-1024",
      "hash": "hash-of-1024",
      "fileSize": "xxxxxxxxxx",
      "dimensions": {
        "value": "1024x768",
        "unit": "px"
      }
    },
    {
      "mimeType": "image/jpeg",
      "fileName": "xl",
      "uri": "ipfs://hash-of-2048",
      "hash": "hash-of-2048",
      "fileSize": "xxxxxxxxxx",
      "dimensions": {
        "value": "2048x1536",
        "unit": "px"
      }
    }
  ]
}

Video Previews

Videos additionally generate optimized video previews where applicable:

max 512w
max 1024w

Like the images, the video hashes get set in formats :

{
  ...
  "formats": [
    {
      "mimeType": "video/mp4",
      "fileName": "md",
      "uri": "ipfs://hash-of-512",
      "hash": "hash-of-512",
      "fileSize": "xxxxxxxxxx",
      "duration": "00:10:46",
      "dimensions": {
        "value": "512x288",
        "unit": "px"
      },
      "dataRate": {
        "value": 320,
        "unit": "kbps"
      }
    },
    {
      "mimeType": "video/mp4",
      "fileName": "lg",
      "uri": "ipfs://hash-of-1024",
      "hash": "hash-of-1024",
      "fileSize": "xxxxxxxxxx",
      "duration": "00:10:46",
      "dimensions": {
        "value": "1024x576",
        "unit": "px"
      },
      "dataRate": {
        "value": 320,
        "unit": "kbps"
      }
    }
  ]
}

Next Steps

As a first pass, creators will have to manually upload a cover image for media other than image and video. This can be automated for other media at a later stage.

[tarwin]

That's super awesome work @pichiste ! My only comment is that not all images / videos are square. Is the idea to crop? Or should this be a max width thing and take the height from that?

[pichiste]

@tarwin Thanks for the feedback, I was thinking of the sizes as max dimensions, so if it's 320x320 the image would be scaled so the longest side is 320px. I do realize though that usually for web the standard is to base it on the width. I'm up for either approach, any preference?

[veqtor]

Why the 320, 640 and 1280, a lot of AI stuff is 256/512/1024, so we'll end up with some aliasing artifacts with these resolutions I think

Also, the different resolutions should be in the formats array, not in attributes, attributes are specifically for like game-items and such, whereas formats are meant to contain different versions of the same media.

https://tzip.tezosagora.org/proposal/tzip-21/#formats-array

[pichiste]

@veqtor thanks for the feedback. Yes you're totally right about formats field! I must have misinterpreted the spec, thanks for catching that. Have gone ahead and updated the draft above using formats, as well as the AI/GPU-friendly sizes, as the previous ones were pretty arbitrary.