webflow
Huge dependency size and tree
This simple library to talk to an API is somehow 113 dependencies long, 10 MiB in size with 5395 files.
└─┬ [email protected]
├─┬ [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├─┬ [email protected]
│ │ │ │ ├── [email protected] deduped
│ │ │ │ └── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ └─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected]
│ │ │ ├── [email protected]
│ │ │ ├─┬ [email protected]
│ │ │ │ ├── [email protected] deduped
│ │ │ │ └── [email protected] deduped
│ │ │ ├─┬ [email protected]
│ │ │ │ ├── [email protected] deduped
│ │ │ │ ├── [email protected] deduped
│ │ │ │ └── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected]
│ │ │ └── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ ├─┬ [email protected]
│ │ │ │ ├── [email protected]
│ │ │ │ ├── [email protected] deduped
│ │ │ │ └── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected]
│ │ │ ├── [email protected]
│ │ │ ├── [email protected]
│ │ │ ├─┬ [email protected]
│ │ │ │ └── [email protected]
│ │ │ └── [email protected]
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ └─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected]
│ │ │ └── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ ├─┬ [email protected]
│ │ │ │ └── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └─┬ [email protected]
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ └─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └─┬ [email protected]
│ │ └─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └─┬ [email protected]
│ │ │ ├─┬ [email protected]
│ │ │ │ ├── [email protected] deduped
│ │ │ │ ├── [email protected] deduped
│ │ │ │ └── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └─┬ [email protected]
│ │ │ └── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ └── [email protected] deduped
│ └─┬ [email protected]
│ ├── [email protected] deduped
│ └── [email protected] deduped
├─┬ [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├─┬ [email protected]
│ │ │ │ ├── [email protected] deduped
│ │ │ │ └── [email protected] deduped
│ │ │ ├── [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├─┬ [email protected]
│ │ │ │ └── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├─┬ [email protected]
│ │ │ │ ├─┬ [email protected]
│ │ │ │ │ ├── [email protected] deduped
│ │ │ │ │ ├── [email protected] deduped
│ │ │ │ │ └── [email protected] deduped
│ │ │ │ └── [email protected] deduped
│ │ │ ├── [email protected]
│ │ │ ├── [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ └── [email protected]
│ └─┬ [email protected]
│ └── [email protected]
├── [email protected]
├── [email protected]
├─┬ [email protected]
│ ├── UNMET OPTIONAL DEPENDENCY encoding@^0.1.0
│ └─┬ [email protected]
│ ├── [email protected]
│ └── [email protected]
├─┬ [email protected]
│ └─┬ [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ └─┬ [email protected]
│ ├── [email protected] deduped
│ ├── [email protected] deduped
│ ├── [email protected] deduped
│ ├── [email protected] deduped
│ └── [email protected] deduped
├─┬ [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ └── [email protected]
│ ├── [email protected]
│ ├── [email protected]
│ └─┬ [email protected]
│ └── [email protected] deduped
└── [email protected]
I've made attempts at this for some open source projects, and what usually happens is some accept PR that replace dependencies for lighter ones, others will often frown or fight every change depending on their internal culture.
So my question is, is there a way to clean this up? And is this repo friendly towards such attempts or not?
Hey @TheThing, thanks a ton for raising this. Some context to set:
- the SDK here is about 95% generated from an OpenAPI spec: https://github.com/webflow/openapi-spec/tree/main
- While this is mostly a library to talk to the Webflow Data API, some extra helper methods are exposed specific for this SDK (this is why
crypto-browserifyis brought in, for webhook verification as an example)
We'll look to see how much of the size we can cut here, but if you have ideas or quick wins you see that you'd like to contribute, feel free to open a PR! Changes to package.json will ultimately come from an internal source-of-truth repo that generates the next SDK output, but we can make those modifications based on any changes you make through an accepted PR
Thank you for the reply.
Yeah the tree is big but that's mostly because of some strange imports by sup-dependency most often.
For example the package form-data is pulling es-set-tostringtag which is pulling a tree of total 23 packages (some deduped but still a too large tree just to do set Symbol.toStringTag, when possible.).
The biggest problem is replacing these (and why I was asking about the culture of the company), is because the only two ways of "fixing" this is:
a) For me to replace a known, big, popular dependency (like form-data) with less-known, less-popular lighter alternatives.
Often frowned upon due to (mis-guided) trust in large/popular dependency or anxiety of some feature breaking due to switching to different dependency.
b) For me to take a large dependency (that does a simple job or is only being used for a simple thing) and rewrite it without the unnecessary sub-dependencies and submitting PR using that.
This is also very highly frowned upon (for a semi-good reason) due to fear of supply-chain attack (who am I, a random person on the internet, to be trusted with having his code be inserted into a large, popular and important library like this).
I've done this before where I replaced a large dependency tree coming from a singular package with an alternative that I worked hard to make and ensure was 100% api compatible, only for it to eventually reach someone higher up in the decision chain reject the whole thing due to fear of supply-chain attack (resulting in a ton of thrown work).
Hence this inquiry and question, I'm just trying to avoid having to go through the motions again and have entire work thrown out due to supply-chain attack fears :)
While this is mostly a library to talk to the Webflow Data API, some extra helper methods are exposed specific for this SDK (this is why crypto-browserify is brought in, for webhook verification as an example)
This is a great example of something that could be cleaned up.
If crypto-browserify is being pulled in for, say hypothetically only to verify HMAC signature, then instead of pulling the entire crypto library that also has a ton of other features (like AES encryption, ellyptic curve and etc), it is possible to replace it with a smaller dependency that only has the exposed functions needed by the SDK and only pull in sub-dependency needed for those.
This is just an example of what can be done to cleanup the tree and something I would happily be willing to do and tackle. I myself have strong fear of supply-chain attacks and so I have big distrust towards large dependency tree, they all look like attack vectors to me.
Unfortunately, in the example I mentioned above, it would probably require me building a new dependency which from a perspective of a developer like yourself, would replace one larger potential supply-chain problem with smaller but focused potential supply-chain problems :)
Hence this inquiry. It would not be hard for me to, for example if I was trying to trim crypto-browserify, figure out what is needed, and then make a dependency that does that to trim of all the other fat. However I would like to avoid doing that work if the fear of supply-chain is stronger than wanting smaller dependency tree and wasting work :)
Hey @TheThing - really appreciate your line of thinking on this, and totally understand where you're coming from in investing time.
Thankfully in these two cases, form-data and crypto-browserify, these are "additive" libraries in that it is not crucial to the generative function for the SDK (and only brought in for some of those extra helper methods I mentioned). In general, we'll first lean on more well-known and maintained dependencies as they are more likely to be patched for security vulnerabilities, bug fixes, all that. In the case of form-data, I think there's definitely room to nix using this. It's only used in this helper method here.
With crypto-browserify, our team didn't find an immediate alternative but we'd be open to moving off this for a slimmer alternative. We'll often run security scanners on our dependency trees to look for vulnerabilities so dependencies may change/evolve. With that in mind, that could mean moving off crypto-browserify (and any other dependency we currently use), leaning in more for future functions we need, etc., and in those cases, wouldn't want your efforts to be thrown away! For more custom control there, feel free to fork the repo and deploy (and simply remove the helper functions if you don't need them, thus removing both dependencies). But would love to still consider any contributions or ideas you might have if you find alternatives.