External APIs

[jimafisk]

I've been chatting through some ideas with @Holben888:

I do think there's a place for dynamic content tho, like supporting Go files instead of JSON files in /content. These could export a function that gets executed at build time and returns content as JSON. Still think plugins at the CMS level are the way to go for popular APIs (Twitter, CodePen, etc.), but it would be nice to be able to manually poll APIs at build time as well!

Couldn't agree more. I don't know exactly how to approach accomplishing this, but I think it's good practice to discuss the experience we want achieve and work backwards to a technical solution.

I tend to agree with Ben and I think the data from external APIs should live in the content/ folder inside a Type just like any other content (vs breaking it out into a separate folder like data or api). The developer would use the field data like any other content and would not have to adjust their workflow. The potential downside is certain folders would be controlled by and API and if a user wasn't aware of this they might make edits to it that could potentially get overwritten on the next build.

It would be nice to be able to simply add "plugins" from the Plenti CLI to integrate with popular third party services. Something like plenti mesh salesforce for a supported plugin or plenti mesh salesforce --from="https://github.com/jimafisk/my-sf-plugin" for community contributed plugins. Not sure if "mesh" is the best keyword, could be: pull, sync, add, integrate, or even extending an existing command like new (e.g. plenti new plugin salesforce).

[awulkan]

Having the ability to pull data from an API at build time would be awesome. It's the main thing I'm looking for right now in the generators I'm considering using. The reason is because I need to pull data from my own backend, while building on sites like Netlify.

Maybe this issue for Hugo can give some inspiration? https://github.com/gohugoio/hugo/issues/5074

[jimafisk]

Thanks for pointing me to that Hugo issue @awulkan, it looks like a good starting point!

I know we don't currently have a graphql data layer, but I also want to look into Gatsby source plugins at some point: https://www.gatsbyjs.com/tutorial/part-five/#source-plugins

[jimafisk]

Go-chi (https://github.com/go-chi/chi) was recommended to me at one point, it looks awesome but might be more targeted building CRUD apps.

Hugo has a similar concept called Data-Driven Content: https://gohugo.io/templates/data-templates/#data-driven-content (also referenced above). An added benefit of not having any reserved keys in Plenti's content source is we don't have to do any "Front Matter mapping."

I'm just thinking out loud how this would work in practice. I would think most people would point at an endpoint that aggregates content of a certain type, and then map each item in that endpoint to a content node for a particular content type in Plenti. If we were pulling from Strapi for instance, we'd just do a GET on /{content-type} and the response would be an array of objects:

Example Strapi response (only contains 1 item)

[
  {
    "id": 1,
    "name": "Restaurant 1",
    "cover": {
      "id": 1,
      "name": "image.png",
      "hash": "123456712DHZAUD81UDZQDAZ",
      "sha256": "v",
      "ext": ".png",
      "mime": "image/png",
      "size": 122.95,
      "url": "http://localhost:1337/uploads/123456712DHZAUD81UDZQDAZ.png",
      "provider": "local",
      "provider_metadata": null,
      "created_at": "2019-12-09T00:00:00.000Z",
      "updated_at": "2019-12-09T00:00:00.000Z"
    },
    "content": [
      {
        "__component": "content.title-with-subtitle",
        "id": 1,
        "title": "Restaurant 1 title",
        "subTitle": "Cozy restaurant in the valley"
      },
      {
        "__component": "content.image-with-description",
        "id": 1,
        "image": {
          "id": 1,
          "name": "image.png",
          "hash": "123456712DHZAUD81UDZQDAZ",
          "sha256": "v",
          "ext": ".png",
          "mime": "image/png",
          "size": 122.95,
          "url": "http://localhost:1337/uploads/123456712DHZAUD81UDZQDAZ.png",
          "provider": "local",
          "provider_metadata": null,
          "created_at": "2019-12-09T00:00:00.000Z",
          "updated_at": "2019-12-09T00:00:00.000Z"
        },
        "title": "Amazing photography",
        "description": "This is an amazing photography taken..."
      }
    ],
    "opening_hours": [
      {
        "id": 1,
        "day_interval": "Tue - Sat",
        "opening_hour": "7:30 PM",
        "closing_hour": "10:00 PM"
      }
    ]
  }
]

To make things easy, do we assume it is always a case that we're pointing at an endpoint that is an array of objects, or is that not flexible enough? If we were to do that, we could just write each item in the array into its own json content file. The integration would be really simple, you could add something like this to plenti.json:

"get": {
  "url": "http://localhost:1337/restaurants",
  "type": "restaurants"
}

This would run at the beginning of the build process and write to the filesystem. That wouldn't take care of fetching any of the referenced images, but maybe it's desirable to serve those from your CMS. Maybe even have optional "destructive" key for if items should be removed if no longer found in endpoint data. It's possible I'm oversimplifying this, I'll need to think on it some more.

[jimafisk]

Strapi example (from above): https://strapi.io/documentation/developer-docs/latest/content-api/api-endpoints.html#get-entries is simple array of objects.

Ghost CMS example: https://demo.ghost.io/ghost/api/v3/content/posts/?key=22444f78447824223cefc48062 starts with {"posts":[

Directus example: https://docs.directus.io/reference/api/items.html. The docs say

The input/output of the API differs greatly for individual installs, as most of the endpoints will return data that's based on your specific schema.

Ponzu example: https://github.com/ponzu-cms/ponzu/blob/master/docs/src/HTTP-APIs/Content.md#endpoints starts with {"data": [

Wagtail example: https://youtu.be/VT9-qdI96rE?t=594 starts with {"items":[

OctoberCMS has plugins that expose individual node endpoints, not sure about aggregate defaults.

Wordpress example from toptal: https://wordpress.org/news/wp-json/wp/v2/posts?per_page=3 is simple array of objects.

Drupal would use Views REST export which is flexible and can be a simple array of objects like this internetdevels example. If using pager_serializer like this d.o example would have a format like {rows: [ though.

Salesforce example: https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/intro_rest_resources.htm appears to be simple array of ojects.

---

So basically my conclusion is we can not assume the base format of items in the endpoints, some include metadata in a separate key, others don't. We need to allow the user to customize this for the specific api they are pulling from. Maybe we could assume the top level is an array of objects, but allow an optional key in plenti.json called "split" or something that allows you to target a specific key that holds the array of items you want to copy into your content source. Do we need to account for other scenarios, like fetching individual items that aren't part of a list?

[jimafisk]

We also need to account for the filename for each node we write to the filesystem. In most scenarios this would be based on a value from the objects in the API. We should have another key called "filename" that points the the key in the API source that we should get the filenames from. We also might want to provide an option to slugify values from the API if the data isn't already formatted correctly for a filename (although we do slugify filenames during build already and soon will be removing spaces as well: https://github.com/plentico/plenti/issues/82). We'd just have to make sure we're comparing the slugified values on updates so things sync up. We could possibly make the "filename" key optional and if it's omitted just use the content type name + an incrementer (e.g. if type is post: post1.json, post2.json, post3.json...) but the challenge would be updates, there would be no way to know which existing file on the local fs corresponds to which object from the API - so maybe it should just be required.

[s-kris]

Would definitely, love to have the feasibility to fetch data from external apis to build 'routes/paths'. Right now, I'm using elderjs, but I'm curious to see the build speeds of go.

[jimafisk]

Thanks for your interest @s-kris, it would be great to get folk's perspectives who are coming from similar frameworks like Elder to know what we're doing well and what could be improved. This issue is becoming a higher priority in our roadmap, so stay tuned to this issue for updates.

[jimafisk]

Just adding thoughts here, we'll have to think through a way to optionally fetch assets (like .pngs) that we actually want to copy to our project vs referencing from an external site.

[s-kris]

Just adding thoughts here, we'll have to think through a way to optionally fetch assets (like .pngs) that we actually want to copy to our project vs referencing from an external site.

This is pretty good option. Not just copying them but optimizing the images for multiple resolutions. Kind of like gatsby-image, next-image, svelte-image components. But it could be added later once core is done due to dev bandwidth.

[BraydenGirard]

Go-chi (https://github.com/go-chi/chi) was recommended to me at one point, it looks awesome but might be more targeted building CRUD apps.

Hugo has a similar concept called Data-Driven Content: https://gohugo.io/templates/data-templates/#data-driven-content (also referenced above). An added benefit of not having any reserved keys in Plenti's content source is we don't have to do any "Front Matter mapping."

I'm just thinking out loud how this would work in practice. I would think most people would point at an endpoint that aggregates content of a certain type, and then map each item in that endpoint to a content node for a particular content type in Plenti. If we were pulling from Strapi for instance, we'd just do a GET on /{content-type} and the response would be an array of objects: Example Strapi response (only contains 1 item)

To make things easy, do we assume it is always a case that we're pointing at an endpoint that is an array of objects, or is that not flexible enough? If we were to do that, we could just write each item in the array into its own json content file. The integration would be really simple, you could add something like this to plenti.json:

"get": {
  "url": "http://localhost:1337/restaurants",
  "type": "restaurants"
}

This would run at the beginning of the build process and write to the filesystem. That wouldn't take care of fetching any of the referenced images, but maybe it's desirable to serve those from your CMS. Maybe even have optional "destructive" key for if items should be removed if no longer found in endpoint data. It's possible I'm oversimplifying this, I'll need to think on it some more.

What if it looked something like this:

"get": {
    "url": "http://localhost:1337/restaurants",
    "type": "restaurants",
    "singleFileType": false,
    "uniqueId": "id",
    "headers": {
        "Authorization": "Bearer <token>"
    }
},
"get": {
    "url": "http://localhost:1337/events?id=1",
    "type": "event",
    "singleFileType": true,
    "uniqueId": "name",
    "headers": {
        "Authorization": "Bearer <token>"
    }
}

The unique id would be used to indicate the primary key (a unique value) coming back in the data. This could then be appended to the name of the type for file generation.

Example: restaurant-2n38s.json restaurant-39vsd.json restaurant-6948s.json

[slanelb]

I came across editor.js and thought of issue #107. Sounded like a good fit or inspiration for Plenti. Some features:

• It is opensource
• It is free
• It is a block-styled editor
• It returns clean data output in JSON
• Designed to be extendable and pluggable with a simple API