resolveuid silently breaks if api domain is different from edit/www domain (ie not using proxy)

[djay]

Describe the bug

• RESTAPI
  • GET - changes any "url" field from resolveuid to the same domain as api e.g. api.mysite.com
    • code
  • POST - changes any "url" field to resolveuid if same domain as api e.g. api.mysite.com
• volto
  • GET - translates any api domain to frontend domain e.g. api.mysite.com -> www.mysite.com
  • POST - doesn't translate urls from www.mysite.com to api.mysite.com. BUG.

If you edit/render volto on the different domain than the api/backend it will appear to work ok BUT

• save will save as www.mysite.com and not resolveuid since the backend thinks its an external url not internal
• if you view the page with links then it will appear to work fine
• SILENTLY resolveuid is not really working so all the things it gives won't work
  • deletion of linked content warnings won't occur
  • if the content moves then unneeded redirects will happen instead of direct links to the new location
  • if the site domain changes then suddenly all your "internal" links break as they point to the old domain e.g. beta.mysite.com -> www.mysite.com

To Reproduce

1. don't use the proxy middleware when editing and have your edit domain and api domain as different.
2. make a slate link to another page
3. if you have your frontend on yet another domain the url will be on your edit domain

Expected behavior

This will be a local link

• [ ] Volto should translate all urls to the api url before save. A reverse of the flattenToAppUrl translation.

  • Api.formatUrl(path) doesn't appear to be used anywhere currently to do this job

• [ ] volto should have a list of all possible domains the site can be served from. - this will help support multiple frontends, and fix issues caused by the site previously using a different domain

• Other solutions considered

  • RESTApi should accept relative urls and Volto should use those instead.
    • much bigger change but in some ways more sensible.
  • RESTApi has a list of url prefixes it should treat as internal
    • con: reproducing the same config in multiple places. easy to forget.
  • Volto calls a restapi to convert urls to resolveuid urls during editing ensuring they are correct

Screenshots If applicable, add screenshots to help explain your problem.

Software (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Volto Version [e.g. 8.5.0]
• Plone Version [e.g. 5.2.2]
• Plone REST API Version [e.g. 7.0.1]

Additional context Add any other context about the problem here.

[djay]

tips on where the best place to patch this would be appreciated.

[djay]

@davisagli what do you think is the best way to fix this?

[davisagli]

@djay We have a project that is served publicly at www.domain.com (with www.domain.com/++api++ as the API url) but edited at editors.domain.com (with editors.domain.com/++api++ as the API URL). Both API URLs forward to the same Plone backend but with different VirtualHostMonster paths, so the backend recognizes absolute URLs that Volto is sending as starting with the portal URL, and converts to resolveuid links correctly.

I think you're talking about a different scenario where you really want the API URL to be on a different domain instead of proxying to it from the Volto domain, but I'm not sure I understand your scenario yet in terms of why that is important.

[djay]

Your setup you are using the middleware proxy. So the editor domain and the editor API domain are the same. That is why it works in your case.
But I think its a bug to say you have use the proxy. Or at least the documentation isn't written that way.

Volto knows that the API domain is and knows if it's different to the current editor domain. So it should do the work to translate any urls it's saving back to the restapi into a url the API knows is internal.

Why the restspi doesn't just use relative urls I don't know. It would make everything easier. But I guess that decision was made a long time ago and it would be inconsistent to switch output and input to be relative urls.

[davisagli]

@djay In production I'm using Traefik as a proxy, not the Volto middleware. Same end result.

Anyway, I basically agree that it's sub-optimal if the backend needs to be able to produce URLs that match the frontend domain. It's too tightly coupled. But, I'll need to study more on what the best way is to make Volto handle this.

I don't think it's out of the question to make the REST API use relative URLs. Yeah we can't break backwards-compatibility, but perhaps we could add a way to opt in to serializing that way via a particular Accept content-type or other header. (They are already handled during deserialization.) Let's figure out what options we have and what is most pragmatic.

[djay]

@davisagli I rewrote the issue to make it clearer as it is really about the edit domain and api domain being different, not really about the frontend domain.

[djay]

I don't think it's out of the question to make the REST API use relative URLs. Yeah we can't break backwards-compatibility, but perhaps we could add a way to opt in to serializing that way via a particular Accept content-type or other header. (They are already handled during deserialization.) Let's figure out what options we have and what is most pragmatic.

the work involved on the volto end to change the urls to either relative or api urls would be the same I think so thats the first step either way? Maybe relative urls is a longer term change?

One thing I did notice working on the frontend for the hydra demo is that it some blocks already set relative urls for some images. it's inconsistent already.

[djay]

@davisagli there is one more thing that bugs me about the current restapi. it has this magic code that finds any field in any block with the same url and does the translation to a resolveUID url. I don't remember seeing this documented and it seems problematic long term. What if a block needs 2 urls? linkLeft and linkRight? How are we ensuring a block plugin author knows they have to call the field url to get correct resolveuid behaviour? I'm not sure what the solution is to this however because currently there is no other need for the restapi to understand the block schemas so it would be a shame to change that. And it does make more sense for the backend to do the resolveUID handling...

• maybe the save includes hints on the fields with urls?
• or anything that looks like a url is translated?
• or save includes the block schemas? but then you also need a standard containment model which isn't there yet either...

One final thing that isn't directly related is about resolveUID links themselves. I've always thought they should include both the UID and the relative path as a backup for when data gets corrupted and the UID location is lost. But thats a different ticket

[davisagli]

@djay Your last comment has some useful thoughts, but could you please not bring up a new topic in the middle of an existing discussion? (I know it's related but we have enough to keep track of here already.) A new issue in plone.restapi would be a better place to discuss that.

[djay]

@davisagli any thoughts on the best place to solve this? it's created an ongoing problem for some of our clients and it would we would like to work on a fix.

[djay]

@davisagli Thinking more about it

• if volto converts urls to be using the api url then what it really means is that any block that deals with urls needs to do this. so providing a generic function is not enough. any block plugin potentially needs to change. Unless volto also uses so magic to turn any block value with the name "url" that matches one of the internal domians, to use the api domain.
• On the other hand doing it in the restapi you already have the deserialisers doing this and they and the magic "url" stuff so at least thats in one place... but it doesn't know the frontend urls and I don't think it should. Configuring it in two places seems painful.

That only leaves a couple of other options

• pass the domains with the blocks to restapi
• change everything to use relative urls to mean internal and make everyones life easier into the future?
  • but you would still need the site urls in volto just in case someone cut and pasted an absolute url into the link field.... :(

[davisagli]

Here's a hack to try making the REST API generate paths relative to the portal root instead of absolute URLs: https://github.com/plone/plone.rest/compare/main...experiment-relative-urls

I haven't done much experimentation yet to see what works and doesn't work with this in place. Volto is able to display and save pages, but deserialization needs some updates to resolve to UIDs correctly (because the current code uses absolute_url, which is affected by this hack)

[djay]

@davisagli wouldn't that require a big backwards compatibility with the restapi to switch to all urls are now relative rather than full urls using the api domain? I agree that makes for a nicer api but the alternative is to just change volto to convert the urls on save to be api urls (which mirrors what it does now on load) and that seems more achievable?

[djay]

@davisagli I guess what I'm saying is that no matter how the restapi expects the urls, relative or using the restapi domain, there is going to need to be a PR for volto to translate any url to either relative or the restapi domain before save. Shouldn't we do that first and switch to relative urls later?

[djay]

@davisagli another way to fix all this is to make volto call an api to convert a url to a resolveuid url, most likely in the url picker and image picker itself. This would also remove the need for the "magic" restapi code that is looking for fields in blocks called "url" and also speed up save.

[djay]

@davisagli any movement on the best way to solve this?

[djay]

@davisagli I cleaned up the description to hopefully make it clearer