Josef Procházka
Josef Procházka
Btw. `sortedcollections` depends on `sortedcontainers` I used sorted containers here: https://github.com/apify/crawlee-python/blob/master/src/crawlee/_autoscaling/snapshotter.py#L10 even though it is not explicitly mentioned in the requirements (due to `sortedcollections` already including it)
> Please add it to the dependencies, you never know when transitive dependencies will change 🙂 Here it goes https://github.com/apify/crawlee-python/pull/1083
There are still some unresolved issues with `apify run` that will manifest when not using `src` folder. Those should be resolved before we change templates https://github.com/apify/apify-cli/issues/766 https://github.com/apify/apify-cli/issues/796
Well, I am not myself in rush to do this issue. Main concern was to make apify cli work correctly both for crawlee based templates and the template repo templates....
Based on the [docs](https://psutil.readthedocs.io/en/latest/index.html#psutil.Process.memory_info): Using `uss` might actually underestimate memory usage as it would not count the shared memory at all. Seems like `pss` might be the best approximation of...
To come up with the test was really hard. The test is not nice at all but testing the memory usage estimation is really tricky due to to Python being...
> 1. how does this compare to what we do in the JS version? It looks like it also uses RSS (https://github.com/apify/crawlee/blob/master/packages/utils/src/internals/memory-info.ts#L85-L85) - I'd like to know why it doesn't...
> > Could you stress test it a little locally since the CI is now not running integration tests on fork PRs? > > Out of about 70 runs, this...
Regarding the timeouts. Currently there is no public way of getting the runtime of the actor run. Specifically runtime of last actor run segment, which in the context of this...
Well there seems to be information that can be used after all. [configuration.timeout_at](https://docs.apify.com/sdk/python/reference/class/Configuration#timeout_at) is actually saving only the timeout of the last run segment and so it should be possible...