Martin Adámek

icon indicating a website link https://mikro-orm.io icon indicating an email [email protected]

icon company @mikro-orm icon location Prague icon location Long time php/nette/doctrine developer, now shifted to javascript world (node/typescript/angular). Building MikroORM in my free time.

Results 768 comments of Martin Adámek

Duplicate Requests Being Processed

The problem here is that the page contains `http` link to itself, you are logging `loadedUrl` which is what you get after redirects, but if you log just `url`, you...

Set desiredConcurrency based on type of crawler and available memory

We already do that based on the type, cheerio sets the default to 10 (#1428). Note that the solution you proposed seems to be quite tied to the apify platform,...

Use priority queue by number of url finds

How do you think this should work? There were some discussions around support for priority in the request queue, but it would still need to be you as the developer...

Use priority queue by number of url finds

Here is the previous issue: #1309, there is some workaround to make this work. Closed as a wontfix, as for us this means implementing such thing on various places to...

Use priority queue by number of url finds

And when would you want to invoke that `priorityFn`? Because to me it sounds like you want that to be triggered when fetching the items, which I don't think could...

Use priority queue by number of url finds

Yeah, but you would want to modify the priority on the already-queued requests, right? That is the part that seems to be impossible to do properly, normally a priority queue...

Use priority queue by number of url finds

Feel free to give it a try, I am not 100% sure if we will want to merge it, but I'll be more than happy to provide more extension points...

Export Dataset to Parquet

Interesting, first time hearing about parquet. I am not sure if we want to support more formats natively, but we could make the exports extensible so you could do it...

Deprecate minConcurrency in Crawler options, replace it with desiredConcurrency

I like the proposal for renaming to `initialConcurrency`, we could rename it right ahead while keeping the old name for BC and clean this up in next major. Btw the...

crawler fail on second run (env CRAWLEE_PURGE_ON_START not used?)

Purging works as expected, the problem here is its rather internal API that is not supposed to be working as you guys are trying to use it - crawlee purges...