spaCy icon indicating copy to clipboard operation
spaCy copied to clipboard

[WIP] Split up `universe.json`

Open koaning opened this issue 2 years ago • 6 comments

Description

This is a work-in-progress PR with a proposal on how to split up the universe.json file. This PR is nowhere near ready to be merged, but I made a PR so that folks may review and hopefully agree on the proposed approach.

Instead of worrying about Gatsby plugins/configurations, I figured I'd just write a node script in the gatsby-config.js file that reads in all files in a folder and appends them to the universe json blob. @ines if you agree that this is an appropriate approach then I can implement this for all the universe items and I can start rewriting the documentation.

Types of change

This change will allow users to contribute their spaCy plugins for our docs without everything needing to go into a single file. That should make it easier to maintain in the long run.

Checklist

  • [x] I confirm that I have the right to submit this contribution under the project's MIT license.
  • [ ] I ran the tests, and all new and existing tests passed.
  • [ ] My changes don't require a change to the documentation, or if they do, I've added all required information.

koaning avatar Jun 17 '22 07:06 koaning

I like the proposed solution! The only thing I'm worried about: Are we going to end up with too many files in the repo? 🤔

ines avatar Jun 29 '22 09:06 ines

What do you mean by "too many files"? The files are in a folder, and it should be well within the limits of git. The main theme to remember is that we want to be strict with regards to the names of said files.

I'll consider this a green light to continue with this PR though. 😄 Thanks for the quick review!

koaning avatar Jun 29 '22 10:06 koaning

Ah. I'm starting to understand what you mean with "too many files". Running the build command seems to freeze GitHub codespaces now. Will investigate!

koaning avatar Jun 30 '22 07:06 koaning

Any ideas about what the issue was with lots of files? I've heard of git having issues with that but I think we have <1000 files (in Universe) and usually you need more than that to cause problems.

This guide to working with repos with many files may be useful.

polm avatar Jul 24 '22 08:07 polm

At the time of making that PR, it was more of a Gatsby issue I think. Not so much a GitHub issue. I should have another look on Monday though, so I set myself a reminder.

koaning avatar Jul 24 '22 08:07 koaning

I was able to build everything in GitHub codespaces, but the preview link wouldn't load. So I tried running it locally, both on my Linux server and on my M1 Macs, but these now give this non-descriptive error when trying to build with Docker:

npx browserslist@latest --update-db
warn Browserslist: caniuse-lite is outdated. Please run next command `npm update caniuse-lite browserslist`
error UNHANDLED REJECTION


  TypeError: Cannot read property 'sourceMessage' of undefined
  
  - error-map.js:15 Object.text
    [spacy-io]/[gatsby]/[gatsby-cli]/lib/structured-errors/error-map.js:15:37
  
  - construct-error.js:30 constructError
    [spacy-io]/[gatsby]/[gatsby-cli]/lib/structured-errors/construct-error.js:30:18
  
  - reporter.js:128 Reporter.error
    [spacy-io]/[gatsby]/[gatsby-cli]/lib/reporter/reporter.js:128:59
  
  - reporter.js:68 Reporter.panic
    [spacy-io]/[gatsby]/[gatsby-cli]/lib/reporter/reporter.js:68:34
  
  - develop.js:96 createIndexHtml
    [spacy-io]/[gatsby]/src/commands/develop.js:96:16

It's unclear if this is a dependency issue or something else. Will try to dive deeper, but any advice is appreciated.

koaning avatar Jul 25 '22 09:07 koaning

Discussed internally - closing for now though we'll follow up on this later.

svlandeg avatar Aug 11 '22 07:08 svlandeg