Dynamic Pages

[Pavel910]

This is

• Feature Specification

Detailed Description

We want to use Page Builder editor as a template builder for data from Headless CMS. Pages need to be rerendered when data in CMS changes, and also when the page template itself is updated.

The #1535 PR is a PoC to verify that it is doable; it contains a lot of code that can now be organized properly. Below is the breakdown of things learned during the PoC implementation and what is necessary to, hopefully, properly implement this feature.

To implement this, we'll make use of Prerendering Service API. There are several triggers that need to be considered:

• PB page template change (this requires all records from Headless CMS to be rerendered for given page template). This makes use of getAll GraphQL query to fetch the necessary records for the respective content model. As there can be thousands of records, this has to be done using pagination.
• Content entry is created/update -> notify PB which will find the template (or templates) used to render entries for the respective content model and add a prerendering task to Prerendering Service.
• Content entry deleted ->notify PB which will find the necessary page template, construct the relevant URL and delete cache from Prerendering Service

Prerequisites

• [x] Enable page data and settings model extension (#1606)
• [x] Add a hook to getPublishedPage CRUD method to be able to handle page not found scenarios using plugins
• [x] Add a hook to getESPageData to be able to expand data stored to Elasticsearch
• [x] Fix security-authorization-anonymous plugin; return early if identity exists (this causes problems if other plugins are registered after this particular plugin)

Once the prerequisites are satisfied, we can start implementing the new feature.

Action Items

• [x] Create api-dynamic-pages and app-dynamic-pages packages to contain all the logic for dynamic pages
• [x] Add internal security plugins which generate a Read API key to allow other apps query HCMS data (see #1535)
• [ ] Add GraphQL data sources get-entry and get-entries to satisfy both directions of changes (see #1535)
• [x] Add LoadDataSources component via pb-editor-content hook, to load data from data sources configured in page settings, and set it into the pageAtom
• [x] Add a <Page> provider to website app so child components can access data sources from anywhere in the tree
• [ ] Create a React component to handle fetching of data from data source. This will ease the creation of individual dynamic components.
• [ ] Add a dynamic set of page builder elements, that bind to the data source (Text, Image, Button, Link, etc.).
• [ ] Add hooks to Headless CMS content lambda to trigger prerendering when content entries data changes
• [ ] Add rerendering of all pages when page template changes.

Bonus

• [x] Research the possibility of adding a smart component wrapper which defines a map between dynamic settings fields and component props (this will be particularly useful for React component generators like Figma).

[bushong1]

Can't wait to see this added!

[webiny-bot]

This issue is stale because it was opened 120 days with no activity. Remove the "stale-issue" label or leave a comment to revive the issue. Otherwise, it will be closed in 7 days.

[webiny-bot]

This issue is stale because it was opened 120 days with no activity. Remove the "stale-issue" label or leave a comment to revive the issue. Otherwise, it will be closed in 7 days.

[ChevronTango]

Will it be possible in the future to support other third party data sources? We make use of appsmith as a lot of our Legacy APIs cannot be migrated into a tool like webiny, so Rest calls, DB queries and elastic search are all things we'd love to integrate with.

I'm also keen to know if this will include client side loading, rather than prerendering everything. I suspect that will be harder to implement, but we'd very much like to have more dynamic content without prerendering, so we can potentially build more responsive UIs, eg adding or viewing comments on posts.

[Pavel910]

@ChevronTango I'll answer the second part of your question first: the main use-case is to prerender full pages, but that doesn't mean that you can't have dynamic content on those pages, since in our Page Builder, every element is a React component. So when prerendering happens, you can control what you render in your components: placeholder content, nothing, etc. Then when your prerended page is served to your visitors, React rehydration happens, and your components run again, so you get a chance to load content asynchronously. There will be a lot of flexibility.

As for external data sources, we don't plan to have those as first-class citizens, at least not for the initial implementation of Dynamic Pages. With our own Headless CMS, we know the data model, so it's easy to provide nice UI which guides the user during content creation, provide UI from custom plugins, etc. With external data sources, since they can be anything, we don't know the data model, so it's a lot harder to manage.

We have a connection between pages, page templates, blocks, CMS content models, so it's all one big ecosystem, where apps play well with each other. I'm not saying we never will support 3rd party data sources, but they're not a priority at this point in time.

[ChevronTango]

Thanks @Pavel910 ! That's super insightful. The REST API is really the only external data source i care about personally, but others might want more. I think being able to call things client side, including our legacy APIs, may help us with our migration and integration, though until the dynamic content feature is releases it would be difficult for me to predict how well that works in practice. I'm curious to see if I can call out to an existing API endpoint and then populate a list component with the results, and for each item, link to a page with more info on it. A fairly standard blog/wordpress/shop style use case. How that works with templating and repeated content i don't know, and is it easy enough that all my teams can make use of it without needing to create a ton of plugins...

I look forward to seeing the release and experimenting with it.

[Pavel910]

@ChevronTango Just to make sure I understand your requirements: do you need to generate entire pages from remote data, or just load this data in custom elements (React components), which are part of the page created within Webiny?

[ChevronTango]

@ChevronTango Just to make sure I understand your requirements: do you need to generate entire pages from remote data, or just load this data in custom elements (React components), which are part of the page created within Webiny?

I suppose I consider those the same thing. How it happens behind the scenes is immaterial so long as the dev can configure a page to display data sourced from an API (via one or more calls), without needing to write any plugins or code.