Fetch Eleventy plugins list from npm API instead of collection of data files

[pdehaan]

I've used npms's API before and enjoyed it.

https://api.npms.io/v2/search?q=keywords:eleventy-plugin+not:deprecated&size=250

We'll have to paginate+offset results if we expect more than 250 plugins since 250 is the max per-page allowed. If we need more data from npm for a package, we can always use npm's own pacote.

But that might be a reasonable enough start and just refresh that via serverless every 24h or whatever.

Other metaphorical wrench in the plans is that everything npm related (from either API) would always just refer to an npm username, which means it might be trickier to display a list of an author's eleventy-plugins on their authors page; unless we try and maintain a lookup table between twitter/github/npm accounts.

---

Fun fact, I was starting to build an eleventy-plugins.dev style site at one point which was just going to deploy daily on a cron job. The one feature I was working on but didn't get around to finishing was a list of keywords/tags or something to make it easier to search/filter by. For example, if I want an 11ty plugin for PWAs, it'd be nice to filter or just do a Ctrl+F on the page for some keywords. I did find a few packages have some weird keywords so I was working on a list of keywords I wanted to ignore (along w/ 11ty, eleventy, eleventy-plugin).

[hmzisb]

Have you been able to fixed this?

[pdehaan]

@hmzisb oh, I don’t think this was a bug, I just saw @zachleat had mentioned it under https://github.com/11ty/11ty-community#stretch-goals-for-later and figured I’m chime in with my unsolicited $0.02 since I prototyped this once upon a time.

I actually was looking at this today though, but couldn’t figure out how to so complex queries using a third party npms client, but it was trivial enough to use axios to hit the api directly.

---

UPDATE: If you're curious what I was thinking, this is a rough attempt at … something. Not sure if I like it yet. It doesn't support paginating the API yet, but we only have ~154 plugins and the max per-page is 250, so we don't need to stress about it yet.

import axios from "axios";

const npmsClient = axios.create({
  baseURL: new URL("/v2/search", "https://api.npms.io").href,
  timeout: 2000,
});

const res = await searchNpms(["keywords:eleventy-plugin"], 15);
console.log(JSON.stringify(res, null, 2));

async function searchNpms(modifiers = [], size = 250, from = 0) {
  // Default to "not:deprecated" if there isn't already a "*:deprecated" modifier.
  if (modifiers.includes((item) => !item.endsWith(":deprecated"))) {
    modifiers.push("not:deprecated");
  }
  const params = new URLSearchParams();
  params.set("q", modifiers.join(" "));
  params.set("size", size || 250);
  params.set("from", from || 0);
  // Fetch ahoy!
  const res = await npmsClient.get("", { params });
  const plugins = res.data.results.map((plugin) => {
    plugin.package.date = new Date(plugin.package.date);
    if (plugin.package.scope === "unscoped") {
      // `scope:"unscoped"` is arguably weird, default to `scope:undefined`.
      plugin.package.scope = undefined;
    }
    return plugin;
  });
  return parsePlugins(plugins, res.data.total);
}

function parsePlugins(plugins = [], total = plugins.length) {
  return plugins.reduce((acc = {}, plugin = {}) => {
    const ignoreKeywords = [
      "11ty",
      "11ty-plugin",
      "eleventy",
      "eleventy-plugin",
      "plugin",
      // Some authors seem to add a keyword w/ their username/scope.
      plugin.package.scope,
    ];
    plugin.tags = plugin.package.keywords.filter(
      (keyword) => !ignoreKeywords.includes(keyword)
    );
    // Is this an official or community plugin?
    const key = plugin.package.scope === "11ty" ? "official" : "community";
    acc[key].push(plugin);
    return acc;
  }, {
    total,
    official: [],
    community: [],
  });
}

[pdehaan]

Only because I find this interesting, here's a self-todo for myself for when/if we do switch from a manually curated list of plugins (circa https://github.com/11ty/11ty-website/tree/main/src/_data/plugins) to something that fetches from npm or npms directly. Here's the current list of plugins from https://www.11ty.dev/docs/plugins/#community-contributed-plugins that do not set a package.json keyword of "eleventy-plugin":

{ npm: 'eleventy-favicon',
  keywords: [ 'eleventy', 'favicon', 'svg', 'png', 'ico' ] }
{ npm: 'eleventy-filter-npm-package-downloads',
  keywords: [ 'eleventy', 'filter', 'npm downloads' ] }
{ npm: 'eleventy-nbsp-filter',
  keywords: [ 'eleventy', 'liquid', 'filter', 'spaces', 'nbsp' ] }
{ npm: 'eleventy-plugin-babel',
  keywords: [] }
{ npm: '@mightyplow/eleventy-plugin-cache-buster',
  keywords: [ 'eleventy', '11ty', 'cache', 'performance', 'resource' ] }
{ npm: 'eleventy-plugin-cloudinary',
  keywords: [ 'eleventy', 'cloudinary' ] }
{ npm: '@pcdevil/eleventy-plugin-intl-utils',
  keywords: [ '11ty', 'blog', 'eleventy', 'internationalization', 'static-site', 'website' ] }
{ npm: 'eleventy-plugin-lazyimages',
  keywords: [ '11ty', 'eleventy', 'plugin', 'lazy', 'lazyload', 'image' ] }
{ npm: 'eleventy-plugin-meta-generator',
  keywords: [ 'eleventy', 'meta tag', 'generator' ] }
{ npm: 'eleventy-plugin-page-assets',
  keywords: [ 'eleventy', 'plugin', 'assets', 'copy', 'resolve' ] }
{ npm: 'eleventy-plugin-pdfembed',
  keywords: [] }
{ npm: 'eleventy-plugin-plantuml',
  keywords: [ '11ty', 'eleventy', 'plantuml', 'plugin', 'diagram', 'markdown', 'highlighter', 'share' ] }
{ npm: 'eleventy-plugin-purgecss',
  keywords: [] }
{ npm: 'eleventy-plugin-reading-time',
  keywords: [ 'eleventy', 'plugin', 'reading', 'time', 'word', 'count', '11ty' ] }
{ npm: 'eleventy-plugin-recent-changes',
  keywords: [] }
{ npm: 'eleventy-plugin-respimg',
  keywords: [ 'eleventy' ] }
{ npm: 'eleventy-plugin-responsive-images',
  keywords: [ '11ty', 'eleventy', 'cloudinary', 'responsive', 'responsive-image', 'responsive-images' ] }
{ npm: 'eleventy-plugin-sharp-respfigure',
  keywords: [ 'eleventy', '11ty', 'figure', 'responsive-image', 'images' ] }
{ npm: 'eleventy-plugin-sharp',
  keywords: [ 'eleventy', '11ty', 'sharp', 'image', 'img', 'transform', 'resize', 'responsive', 'picture', 'srcset' ] }
{ npm: '@resoc/eleventy-plugin-social-image',
  keywords: [ 'resoc', 'eleventy', 'social' ]}
{ npm: 'eleventy-plugin-toc',
  keywords: [ '11ty', 'eleventy', 'plugin', 'toc', 'table of contents' ] }
{ npm: 'eleventy-plugin-typeset',
  keywords: [ 'eleventy', 'typeset', 'plugin', 'typography', 'punctuation', 'quotes', '11ty' ] }
{ npm: 'eleventy-xml-plugin',
  keywords: [ 'eleventy', 'plugin', 'rss', 'sitemap', 'xml', 'dates', 'escape' ] }
{ npm: '@shawnsandy/npm_info',
  keywords: [ 'npm', 'filter', 'eleventy', '11ty' ] }

Should be trivial enough for me to fire off 24 upstream PRs. A bit more quick sleuthing shows 23 of the 24 repos have a listed repository, and all 23 were on GitHub; that 24th plugin, eleventy-plugin-babel, is also on GitHub, it just didn't list a repo on npm.

---

UPDATE: Alrighty-o, submitted 23 PRs which added "eleventy-plugin" to keywords[] in their respective package.json files. Realistically I expect about 50% success/merge rate, most looked untouched in a long time, and one repo was set to read-only, so it rejected my PR at the last step.

[pdehaan]

OK, I actually built my thing. "Look on my UI, ye Mighty, and despair!" https://github.com/pdehaan/eleventy-plugins

Actually was a slightly bigger challenge than I thought, although not sure we'd need an author archive page or keyword archive page or most of the stuff I tried shimming in there for a fun weekend project. But I did figure out how to do my recursive, promise-based API fetcher to deal w/ pagination limits. Currently npms.io has a limit of 250 records per request, which isn't an issue for "eleventy-plugin" yet (~163 results; 1 request), but I think "markdown-it-plugin" had ~517 results (or 3 requests), and "eslint-plugin" had ~2056 results (or 9 requests).