Jim Smart

Hi, the repos mentioned in https://github.com/gocolly/colly/issues/103#issuecomment-364816457 (above) contain my prototype code. Feel free to open issues on those repos if you have any questions — happy to help, as best...

@hsinhoyeh For our use case, scraping many hundreds of thousands of HTML files (with a big variety in sizes), there is certainly a benefit to storing the files in SQLite,...

Excuse me for asking, I know this isn't really the correct place (though arguably it might be the best place), but how can I get involved with / on a...

Arguably, if the caller specifically asked for N-Triples, the library should really only output valid N-Triples. — So perhaps the writer should be able to enforce that? Just my 2p!

Sure, understood. I can see both sides. — Maybe it just needs more/improved documentation? (Certainly no perf impact with this option! 😉)

Another possibility would be: when the caller asks specifically for N-Triples, swap out the existing code that writes quads, for a function that only writes triples (completely ignoring the graph)....

Ok, well that’s your call. I was just offering a solution that I hadn’t seen being discussed yet.

That’s a good idea!

Hi, I see an issue filed to do with mistakenly treating a file as incomplete when it was actually complete, and I see here you are trying to ascertain whether...

I think it certainly requires more thought / discussion, but I currently think hooks are probably the way to go. It's possible that the existing BeforeCopy might work for me,...