Dropped request handling

[Ziinc]

There needs to be a mechanism to handle dropped requests/error response codes. Would it be better to implement it through a callback in the settings, or through a pipeline?

[Ziinc]

To clarify, when I say handle, I mean "to do something with the request, like update a database)

[dkgitdev]

I suggest you moving onto Request / Response Middlewares.

This way we can integrate current middleware type as Request Middleware and provide a way to basically tag, filter or do something else upon getting new response using Response Middleware.

[Ziinc]

Hmmm. This would mean a middleware piping step prior to piping the success response through the item pipeline.

Let's call this hypothetical step Response Middlewares.

Should the sole responsibility of the Response Middlewares be to do something with the responses, successful or not? This could be the step where we check for retries, and tag responses, or basically control the flow of the response (whether to be dropped, logged, piped etc).

Currently, as is, only code 200 responses are passed to the item pipelines

[dkgitdev]

I think, that the more flexibility you have – the better.

The response middleware list can contain only one middleware by default, like AllowStatusCodes with the tuple of 200, 201 as argument. This way it'll provide both sensible defaults and flexible way to inject some additional code, based on what actions you actually want to happen.

Because when the status code IS the information you'd like to gather, you'll need some way to override the defaults in a non-hacky way.