Consider auto-failing pages that have 5xx HTTP status code

[jancurn]

Currently, these pages are not considered failed, and thus not retried. On the other hand, Cheerio Scraper retries them. We should probably consider 5xx errors as failures and retry.

[fazio91]

Hello @jancurn, how about if we set an array of error codes to consider in the sessionPoolOptions?

Like this:

sessionPoolOptions: { maxPoolSize: 100, errorStatusCode: [401, 403, 429, 500] // optional }

Thank you!

[B4nan]

You can already set the blocked status code via sessionPoolOptions in crawlee:

sessionPoolOptions: { blockedStatusCodes: [401, 403, 429, 500] }