cdnjs

[victorb]

Would love to have a IPFS compatible fork of https://github.com/cdnjs/cdnjs serving files via IPFS. Super large repository though but will give it a try to develop the integration locally.

[jbenet]

cc @lgierth

[davidar]

On hold until ipfs add performance improves, too many small files

[eminence]

Ok well I bit the bullet and added this to IPFS. Here is the result: http://ipfs.io/ipfs/QmZJKsLpebYqHApRLeoLhj2NsXZ2JoXqhSWkxafxaBXYu7

A few observations. Note that for all of the below comments, the daemon was not running, and I was IPFS version 0.3.11-dev

• Performance wasn't as terrible as I thought it would have been, but it was pretty darn slow. Took about 15 or 16 hours to add everything
• When I first added things, I did it on directory at a time: $ ls ajax/libs | xargs -N1 ipfs add -r As expected, this kept memory usage in check (mostly).
• Later, after all the data was in IPFS, I tried added the entire ajax/libs directory at once. This went quite quickly, however memory usage ballooned very quickly to about 75% of my machines available memory (about 24 GB used out of a total of 32GB available). I killed ipfs at this point, and wasn't able to let it run to completion.
• Since I wanted to have a single tree containing everything, and I wasn't able to add the whole directory at once, I had to resort to building up a tree with ipfs object patch, by manually calling add-link for each directory. This was very easy to do, but it was very slow! It appears that this large repo is causing some serious delays when running IPFS commands:

$ time ipfs object new unixfs-dir
QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn

real    0m14.855s
user    0m19.136s
sys     0m0.632s

This means that is took 6 hours to call object patch add-link for every top-level directory. I believe this 15 second delay is just some startup delay, so if I had the daemon running at this point, I believe this process would have been much faster (since there would be no startup delay)

• IPFS deduplication appears to be helping a lot here! Compare the size of the cdnjs/ajax/lib directory to ~/.ipfs :

achin@diax:~/devel$ du -sh cdnjs/ajax/libs/ ~/.ipfs/
22G     cdnjs/ajax/libs/
13G     /home/achin/.ipfs/

I took a look at every single hash to see what ones were duplicated. In total, about 180000 difference hashes were dedups at least once in this tree. The most dedupd hash was a small 1x1 PNG file with 12376 links. The second most deduped file is an empty file with 696 links.

I've uploaded a file that lists all dedups. I haven't done the analysis to see if it would account for the savings reported by du

• Time to pin everything. Not terrible:

$ time ipfs pin add -r QmZJKsLpebYqHApRLeoLhj2NsXZ2JoXqhSWkxafxaBXYu7
pinned QmZJKsLpebYqHApRLeoLhj2NsXZ2JoXqhSWkxafxaBXYu7 recursively

real    7m24.915s
user    5m16.676s
sys     0m32.560s

• I've uploaded a copy of the output of ipfs diag sys for your info

[davidar]

@eminence awesome, thanks for tackling this and writing up the details :)

Cc @whyrusleeping @rht @diasdavid

[eminence]

Second day follow up notes:

• Updating this repo is pretty straight-forward. After fetching the cdnjs repo, and fast-forwarding to the latest master, I can use git diff --dirstat to see just the libraries that changed. This makes it easy to ipfs add -r -q ajax/libs/library_that_was_updated | tail -1 and then ipfs object patch add-link. I've confirmed that the object patch is nice and quick when I leave the daemon running.
• Note that I'd love a mode in ipfs add that just gives me the top-level hash of the thing that I'm adding, so I can dispense with the tail -1 stuff.
• This manual tree management is very doable, but also annoying. I admit I've not been following the Files API stuff -- would it help here?
• To make use of these hosted libraries, I've added them to IPNS here: http://ipfs.io/ipns/em32.net/archives/cdnjs

But IPNS is still too slow. Or rather, it is not consistently fast. I wrote a script that requests the same file several times, and records how long it takes. The data for 60 requests is here. The summery is that most of requests are quick (about 0.5 seconds or less). But some requests take 10 seconds, others up to 60 seconds. When your website is loading many different assets (like cdnjs libs), this can result is very noticeable delays.

• I have (anecdotal) evidence that even trying to ipfs get some of these large cdnjs directories can cause run-away memory leaks. This may make it difficult for this repo to be effectively mirrored/pinned.
• I've committed to @whyrusleeping to re-run these tests with the 0.4.0 branch. I'll report back in the coming days

[davidar]

Note that I'd love a mode in ipfs add that just gives me the top-level hash of the thing that I'm adding, so I can dispense with the tail -1 stuff.

I think ipfs add -q does that

This manual tree management is very doable, but also annoying. I admit I've not been following the Files API stuff -- would it help here?

Probably, yes.

[jbenet]

I think ipfs add -q does that

ipfs add -q outputs only hashes. still need tail -n1. but yeah, we should add a flag that outputs only the last root. maybe --only-root ? or -Q, --root-quiet? idk.

[jbenet]

This manual tree management is very doable, but also annoying. I admit I've not been following the Files API stuff -- would it help here?

yes, very much so.

[jbenet]

But IPNS is still too slow. Or rather, it is not consistently fast. I wrote a script that requests the same file several times, and records how long it takes. The data for 60 requests is here. The summery is that most of requests are quick (about 0.5 seconds or less). But some requests take 10 seconds, others up to 60 seconds. When your website is loading many different assets (like cdnjs libs), this can result is very noticeable delays.

yes, ipns is still very slow. there's caching (the fast results), but we need to fix this at the dht query level

[eminence]

After testing with 0.4.0, the experience was much better!

• Memory usage was totally under control, and mostly stable the entire time.
• I was able to add the entire libs directory at once. This took about 9 hours, which is much quicker than before. This isn't a totally fair comparison, since I used different hardware for each test (0.4.0 versus 0.3.8), but even trying to take into account the hardware differences, I believe 0.4.0 was faster
• Rescanning the entire libs directory, after the initial add, took about 50 minutes. This would represent the time needed to do an incremental update. Obviously using the git diff trick from above would improve this, but 50 minutes seems pretty acceptable to me.

If anyone is running a 0.4.0 node, the result is here: QmRnvPSCNmYHdYQAo6JUWJPW8uVQv7z6D9nSQmw5qbHVWy

[jbenet]

Good news. Though still waaaay too slow for my liking.

Adding concurrent add will help here. I believe we have not added this. On Tue, Dec 15, 2015 at 09:08 Andrew Chin [email protected] wrote:

After testing with 0.4.0, the experience was much better!

Memory usage was totally under control, and mostly stable the entire time.

I was able to add the entire libs directory at once. This took about 9 hours, which is much quicker than before. This isn't a totally fair comparison, since I used different hardware for each test (0.4.0 versus 0.3.8), but even trying to take into account the hardware differences, I believe 0.4.0 was faster

Rescanning the entire libs directory, after the initial add, took about 50 minutes. This would represent the time needed to do an incremental update. Obviously using the git diff trick from above would improve this, but 50 minutes seems pretty acceptable to me.

— Reply to this email directly or view it on GitHub https://github.com/ipfs/archives/issues/35#issuecomment-164775232.

[eminence]

For me, the gold standard of merkledags is probably git :) So I timed how long it takes to add this directory tree into a new git repo -- 41 minutes! About an order of magnitude faster than IPFS.

[whyrusleeping]

@eminence yeah, we've got a little ways to go still, but keep in mind that git uses a faster hashing algorithm, and doesnt chunk objects. I have one more changeset to apply that should get close to leveling the playing field. Just have to polish it a bit.

[rht]

In some cases, e.g. lots of 1MB files instead of 1KB files, ipfs is way faster than git https://github.com/ipfs/go-ipfs/pull/1973#issuecomment-157738243

[rht]

git uses a faster hashing algorithm, and doesnt chunk objects.

The chunking equivalent in git would be git gc, but only after several revisions?

[jbenet]

we should consider moving the default to https://blake2.net/ it's designed for this.

[rht]

That would explain rsync's speed, since it uses blake2. Perhaps it is possible to go ~O(rsync) with this.

It would be more effective, though, to use blake2 into 0.4.0 before its release (combining all the incompatible changes into one release?).

[jbenet]

It would be more effective, though, to use blake2 into 0.4.0 before its release (combining all the incompatible changes into one release?).

blake2 isnt incompatible. https://github.com/jbenet/multihash :)

[jbenet]

oh i guess it is, because it's not included in the 0.3.x codebase, right.

[jbenet]

yeah it would be nice, but idk if we can land it in time. @whyrusleeping wants to ship 0.4.0 soon. would be nice to add blake2 and ipld support, but not sure if we'll get there in time

[eminence]

Even though git is super fast here, ipfs isn't unusablely slow. as an end user, i have so many more things on my wishlist that are more important to me than speed

(edit: but i do indeed understand the desire to put all breaking changes into 1 release)

[davidar]

@rht What percentage of the total runtime is currently consumed by the hash function?

[rht]

...I did instead with testing dev0.4.0+blake2b. There is more cpu consumption and a speedup but not noticeable due to the jitter like in https://github.com/ipfs/go-ipfs/pull/2039#issuecomment-164978328. Perhaps it could be significant once dev0.4.0 goes to ~O(git) or ~O(rsync). Though I could have just checked the runtime percentage of the hashing.

Currently, both ipfs and git are slow for adding large things. edit: s/moving/adding/

[Kubuxu]

I re-archived it on my own. Available under fs:/ipns/cdnjs.ipfs.ovh

Notes: It took about 6h, I had one crash during that time.

[xioustic]

Anyone interested in a deterministic, efficient node script that wraps the command-line to do the git pull / git diff / ipfs add / ipfs object patch manual work that does this into the existing index you last put into the IPFS network? Could also handle the memory issue (restarting the daemon or sets of potential offending processes when they get near a threshold).

About four months ago I did this and I'll need to dig up my work sitting on one of my VPSes; simple dumb 'ipfs add -r' every time I got pulled was quite expensive. Might be reusable in other related archive targets (other big git repos that are more archive repositories than single codebases). Just requires 'git', 'ipfs' and 'node' on the path of the indexing machine. Maybe someone has already done this or perhaps the ipfs internals have improved enough to remedy the need for this?

I just found out about this "IPFS Archives" project at the Decentralized Web Summit, pretty exciting. @whyrusleeping still doing the workshop on IPFS Archives and Versioning, so tagging him.

[Kubuxu]

Me and @Magik6k are working on adding the cdnjs currently.

I added whole cdnjs some time ago, it is available under fs:/ipns/cdnjs.ipfs.ovh but I think most of that data is currently gone (due to problems with local IPFS repo).

Adding cdnjs is nice stress test for IPFS: https://github.com/ipfs/go-ipfs/issues/2823 https://github.com/ipfs/go-ipfs/issues/2828

[Kubuxu]

Note about publishing the cdnjs:

We shouldn't use /ipns/ to reference the cdnjs in HTML. IPNS currently doesn't work with browser level caching (no HTTP 304 response code if path is IPNS path).

[notslang]

Updated CDNJS hash: https://ipfs.io/ipfs/QmPJnEf5933cXteZmaMJkphCW1CtpcMMVx7N6rUr8cZAok And a little script for generating it, in case anyone else wants to try:

#!/bin/bash
HASH=$(ipfs object new unixfs-dir)
for FILE in cdnjs/ajax/libs/*; do
  LIB=$(basename $FILE)
  echo "adding $LIB"
  LIB_HASH=$(ipfs add -r -H -q "$FILE" | tail -n 1)
  HASH=$(ipfs object patch $HASH add-link $LIB $LIB_HASH)
done
echo "final hash: $HASH"

...However, this doesn't handle symlinks, which do exist in CDNJS. I've gotta decide how to deal with those.

[eminence]

What needs handling exactly (with regard to symlinks)?

[notslang]

If you click on one right now (like https://ipfs.io/ipfs/QmPJnEf5933cXteZmaMJkphCW1CtpcMMVx7N6rUr8cZAok/zocial) it's broken, whereas https://cdnjs.cloudflare.com/ajax/libs/zocial/1.2.0/css/zocial.css works.