Locally uploaded images inconsistently loaded as page covers and hero-icons

[jmuchovej]

Description

I've been using a private Notion page to back my site – retrieving blocks with NotionAPI({ activeUser, authToken }), but had to modify createPreviewImage({...}) to forward cookies (notion_user_id and token_v2) in the got request to retrieve LOIP preview images.

While I can now retrieve the LOIP previews on the page covers / hero images, they never completely load. What's odd is that the same scenario (locally uploaded image) works perfectly fine as a page content block.

Is this a limitation of next/image, expected behavior, or maybe something else? 🤔

Notion Test Page ID

N/A because this strictly occurs with the Notion pages set to private.

Sorry for the sparse issue. 😅 I mainly wanted to sort out whether this is known to be a problem or not.

I saw #246, but I don't think this is a dupe. I'm definitely not running into an issues with extensions/whatnot. (I've tried in freshly installed browsers.)

I also saw https://github.com/NotionX/react-notion-x/issues/232, which could be the source of this problem, but doesn't seem too likely since uploaded images can clearly be retrieved/processed/displayed as content blocks.

Sooo... maybe it makes sense to reframe this as a question of: "Are page hero images / page covers retrieved differently than content blocks?" (or something similar).

[transitive-bullshit]

I haven't tested very much with private pages, but I have a theory of what might be happening.

Notion uses AWS S3 private files to host any UGC assets / images. When we fetch info about a page and its blocks, we ask Notion for a signed version of these URLs (here). The problem is that these signed URLs have a relatively short expiration time. So while they may work at first, with cached SSR pages, these URLs end up expiring when they're served to users.

This is a problem for both private pages and public pages, however, the expiration time may be more strict for private pages (I'm not sure).

This is definitely an issue I'd like to address, though I can't be 100% sure it's what you're experiencing.

Are page hero images / page covers retrieved differently than content blocks?

I don't think so, but I could be wrong. They use different code paths, but are both ultimately rendered using LazyImage and mapImageUrl in react-notion-x.

[jmuchovej]

So, I'm not exactly sure how signed URLs work; but from some quick searches/skimming, it seems that signed URLs allow retrieval of assets/images without needing to forward cookies of any kind. Is that correct?

I'm also not sure how the signed URLs works with LazyImage and mapImageUrl, but do you know what kind of token is generated by the getSignedFileUrls endpoint that's queried?

Essentially, I found this SO answer that alludes to signed URLs being valid for as little as 1s. I'm not sure how Notion has their S3 setup sign these URLs (in terms of how long tokens are valid for), but it seems like tokens could expire pretty quickly (which would explain not being able to retrieve some UGC).

BUT... I'm not sure how relevant this is, as it seems that the code path hero images and page covers take results in a failure to retrieve a signed URL the way that blocks do.

I think I can make some time to look into this, but I've never used react-notion-x and your notion-client, so there's definitely a learning curve there. 😅

[transitive-bullshit]

I think your reasoning is valid so far.

If you want to look into things, there's a contributing guide for getting up & running with local development. Let me know if you have any questions.

[jmuchovej]

So, I only just got around to setting up a dev environment using examples/full in the react-notion-x repository.

What I've found, so far, is that UGC (on private pages, it seems) fails to trigger a retrieval in getSignedUrls. (Essentially, there are no signed_urls present for UGC on private pages.) What's odd is that public UGC seems to correctly trigger retrieval of signed URLs, as you can see in the the default example page (067dd719a912471ea9a3ac10710e7fdf).

I'm looking into why this might be, now; but figured I'd log this in case that's expected behavior.

---

So, I've sorted out why the cover images on Collection Cards don't seem to be showing up – they're never queried for a signed URL.

Something else I noticed is that the blocks contain many file_ids, which I would wager need to be turned into signed URLs; yet none of these files are ever given this treatment. Essentially, for block.type === "page", only the Page Cover is converted to a Signed URL, which means a UGC page icon wouldn't be loaded correctly.

I'm currently sorting out what's going on with that, so that all file_ids present can be converted converted into Signed URLs (as needed).

This leaves me with two questions:

1. Is there anything wrong with trying to retrieve a Signed URL for non-AWS links? (From what I can tell, this strictly wastes computation but is otherwise harmless.)
2. I've noticed that addSignedUrls is not given all of the content IDs, so it doesn't actually retrieve all files needed on the page – so I've been passing in recordMap.block – is that problematic (like, security-wise, or similar)?

Edit: Adding more of a progress update.

[jmuchovej]

So, I've been able to get signed URLs for private UGC working. This will also works on collection cards, page heros, and page icons. (I noticed that these bits strictly work if folks make them public – as no signed_urls are referenced. (Maybe it makes sense to lift this into defaultMapImageUrl?)

I've tested this on both the examples/full set of Notion pages as well as my own private pages.

To do this, I had to parsePageId on a given URL to ensure that page_icon and page_cover were allowed to both be UGC – since the current implementation forbids UGC page icons if it's set to private. I think this shouldn't cause collisions, but I honestly don't know how Notion issues UUIDs.

I'll see if I can wrap things up into a PR for review in the next few days.

Would still appreciate some answers on the above, though. 🙂

[transitive-bullshit]

Thanks @jmuchovej; this is a really great investigation.

Is there anything wrong with trying to retrieve a Signed URL for non-AWS links? (From what I can tell, this strictly wastes computation but is otherwise harmless.)

I don't think there's anything wrong, though I can't be sure since this is all based on reverse-engineering Notion's private API.

I've noticed that addSignedUrls is not given all of the content IDs, so it doesn't actually retrieve all files needed on the page – so I've been passing in recordMap.block – is that problematic (like, security-wise, or similar)?

Much of the implementation is based on trying to see how the Notion API works in practice via dev console usage and then re-creating it locally. The handling of private pages has definitely not been tested nearly as much as the primary use case of public pages, so I can't say much about what Notion's API passes for these types of blocks.

I don't think passing in recordMap.block is problematic at all. It's probably a good call.

So, I've been able to get signed URLs for private UGC working. This will also works on collection cards, page heros, and page icons. (I noticed that these bits strictly work if folks make them public – as no signed_urls are referenced. (Maybe it makes sense to lift this into defaultMapImageUrl?)

I'm not sure exactly what you're suggesting, but in general I'm very open to pragmatic solutions that work and don't break things 😄