traverseda

I'm not a conda user, is there any benefit to making something a conda package instead of a pip package? I've honestly never seen that done before.

So the main reason why you should support pip is so that I can add `pythonocc-core` to my python project's requirements. Where as if this project uses conda, well anything...

It also doesn't seem to work with existing virtualenvs? Or at least I can't get it to work. So that means no pipenv, no poetry, no pipx, etc. Using conda...

Right, so I don't have a lot of experience with build systems, aside from occasionally changing a few flags here and there. My day job and my failing business keep...

What do you mean by monolithic? You saying that we'd need to also make gmesh a package, and link against that?

Any progress with this? It's something I'm finding problematic at the moment.

Looking at this further, write performance *does not* scale linearly with number of indexes. That is a huge problem for large files, and probably causes significant performance issues for larger...

>How big are the WARC files you're dealing with? Working from `enwiki-20200201-pages-articles-multistream.xml.bz2` I have 72G of warc files, split up into 9GB files (have have an 8 core CPU, so...

Huh, I used warcio to create these warc files, and I haven't gotten any errors, so I'd be surprised if the input I was feeding it was the problem. I'm...

So it looks like unsorted CDX files return in a reasonable/expected amount of time. ``` time cdx-indexer ./0.cdxj ./collections/wiki/archive/0.warc 370.75user 7.78system 6:22.46elapsed 98%CPU (0avgtext+0avgdata 34036maxresident)k ```` But sorted CDX indexes...