Allow for custom requests sessions

[michaelweiser]

The requests module internally works with Session objects. These can be customised in behaviour e.g. using retry adapters. This is extremely useful for error handling because the user application doesn't even get to see and need to handle errors until a configurable number of retries with optional backoff has been tried. Unfortunately, the module-level API of the requests module does not allow to supply such a customised Session object. Instead it creates a new Session for each request and needs to use it as a context handler to make sure it leaks no sockets - see https://github.com/psf/requests/blob/143150233162d609330941ec2aacde5ed4caa510/requests/api.py#L57 for details. The module-level API is a very thin shim which direcly hands all requests to this per-request Session object.

This change switches the Cortex API layer to use a Session object for requests directly and adds a parameter so users can provide their own customised Session object.

As a side-effect, this switches the module to using a persistent Session object that is re-used across requests. This should be more efficient in that it can re-use an established connection and potentially connection pooling.

Addendum:

• see https://github.com/scVENUS/PeekabooAV/pull/175/commits/896423636686b461f857b2cb32fc21c15f31126f for this functionality in action
• it does not appear to me that cortex4py makes any assumptions about requests being isolated and their sessions temporary. Any feedback on whether this change would break any assumptions about module idempotence would be appreciated.

[Jack28]

I have tried this out and discovered that when submitting requests with a session cookie, the API also requires an XRSF token. The WebUI learns this token from the /api/user/current endpoint that can be accessed with just a session cookie but without an XSRF token. cortex4py without this PR does not run into this problem because it starts a new session on every request by submitting the bearer token and never learning any session cookie from the response. Our requests session object however will learn the session cookie automatically from any request and there's no official way to reset the session cookie store.

We have tried to learn the cookie the same way the web ui does which seems to work but requests are still rejected by the API. Reason unknown.

But it turns out that the requests module has what seems to be a bug that setting the session cookie to any value will effectively mask the one returned by the server in following requests. Use that workaround to the workaround for now.

I've implemented this workaround for now: https://github.com/michaelweiser/PeekabooAV/commit/f5f5e9dc789632042a1deaf37fbf50ddf6b78de3

What is the recommended way to proceed here? How should the API treat the XSRF token and SESSION cookie?

[michaelweiser]

I have changed the patch so it leaves the default behaviour of cortex4py alone. See https://github.com/scVENUS/PeekabooAV/pull/175/commits/fafa69b47df06ec0c8ab7d0453196778af9be97d how the user can provide a forgetful cookie policy to avoid any possibility of confusing Cortex.

I would mention that while I remember to have reproduced the issue of Cortex setting a SESSION_COOKIE before handing us a CORTEX-XSRF-TOKEN and then refusing to talk to use as described above, I can no longer, neither with the training vm 4.0 nor the docker container image. Good news, I'd say.

[michaelweiser]

Ah, I see another patch went in here as well, dealing with an annoying AttributeError exception when trying to run a nonexistant analyzer. Let me know whether this should get its own PR.

[michaelweiser]

Any news on this? I still have a need for this functionality.