Eric Valls comments

Results 9 comments of


                                            Eric Valls

Crawlee increasing concurrency until it dies

I sent you a discord private message as I'd rather not share too many details here

Crawlee increasing concurrency until it dies

Oh I see, I was assuming it was more like a percentage. But, either way, if my pod is being killed because it is using more than (or close to)...

Crawlee increasing concurrency until it dies

I am not sure it is related, but I am not overriding available_memory_ratio default value in the configuration, so shouldn't crawlee target to using 25% of the available memory?

Crawlee increasing concurrency until it dies

I am not specifying it either 🤔

Updating request queue metada performs full table scan in SQL storage

I added this index and there is no table scan anymore ``` CREATE INDEX idx_request_queue_records_queueid_handled ON public.request_queue_records USING btree (request_queue_id, is_handled) ``` although (without having no idea on what the...

Updating request queue metada performs full table scan in SQL storage

oh I see, and what do you think about adding the index?

Updating request queue metada performs full table scan in SQL storage

Perfect, thanks! I am also having a similar problem with the datasets. Adding a new entry triggers an update in the datasets table (only metadata) and it also blocks my...

Updating request queue metada performs full table scan in SQL storage

what would you think about calculating these values for RequestQueue on demand? Is it possible that these counts are much more written than read? The change could be minimal, just...

Updating request queue metada performs full table scan in SQL storage

Thanks!! We can be beta testers if you need it