Check each subcomponent of the pathname against deny list

[alanshaw]

Currently we check the CID is not on the deny list, but it's possible for a CID + path to be on the deny list, meaning that anything below a given directory could be blocked and our implementation would not catch it.

The majority of items on the badbits list (and all items currently on our list) are CID only and the badbits automation only adds entries for CIDs (i.e. not CID+path). Ideally we'd check each component of the path against the deny list however this would involve many KV requests for a long path and we have only 50 sub-requests max for each worker invocation.

Some options:

1. Bundle deny list with app and keep in memory (short term because there is both a bundle size limit and memory limit in CF).
2. Support paths up to 6 deep (a worker can send 6 concurrent requests so this should avoid any slowdown, but would potentially not catch 100% of blocked paths).
3. Support up to ~40 paths (diminishing returns and slower response times - still may not catch 100% of all blocked items)
4. Something different entirely.

[thattommyhall]

BTW, I spent a fair bit of time making sure I can re-process the existing denylist. I am not that happy that we chose to concatenate the path and the CID before hashing so we could agree something else too.

The discussion was https://github.com/protocol/badbits.dwebops.pub/issues/40