aiohttp
aiohttp copied to clipboard
aio-libs
Implementing auth handlers with `aiohttp.client`
Is there a preferred way to hook in auth handlers to the HTTP client? If not, can we specify one, maybe with code changes to ClientSession to support it?
Implement e.g. OAuth, Negotiate, Digest auth on the client side, instead of just supporting Basic.
More generally, have some way of hooking in error handlers for 4xx responses that can perform additional requests before returning a response. Could also be used to transparently back off on 420 and 429 responses.
My thinking was putting something near the bottom of the while True loop in ClientSession.request to allow something to be hooked in to handle 4xx (or any other) status by manipulating the request and performing it again. Nowhere near sure on the details yet though.
Custom authentication mechanisms in requests would be awesome. Right now it looks like I have to subclass the ClientRequest and overload update_auth to perform my custom authentication logic.
If we simply didn't have the expectation that an auth was a BasicAuth but something that responded to an encode method (or something similar) that would be awesome. I'd also really like it if the request itself could be available for the encoding process for mechanisms that want to have a signature of the request itself.
Any progress on this one? Digest auth is often needed to interface with IoT devices/IP cameras.
Would this be a good place to mention I'd like to add support for AWS4Auth?
Would it make sense to add event hooks similar to the requests package? http://docs.python-requests.org/en/master/user/advanced/#event-hooks This could allow devs to register custom authentication methods.
IE:
class MyAuth:
async def do_custom_auth(self, response):
if r.status_code == 401:
# do custom
def __call__(self, request):
request.register_hook('response', do_custom_auth)
That would be very cool.
On Fri, Dec 1, 2017 at 8:25 AM, dadocsis [email protected] wrote:
Would it make sense to add event hooks similar to the requests package? http://docs.python-requests.org/en/master/user/advanced/#event-hooks This could allow devs to register custom authentication methods.
IE:
` class MyAuth: async def do_custom_auth(self, response): if r.status_code == 401:
do some custom auth
def call(self, request): request.register_hook('response', do_custom_auth)
`
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/aio-libs/aiohttp/issues/434#issuecomment-348493843, or mute the thread https://github.com/notifications/unsubscribe-auth/AClF3K2xBosTHiayvQqEdRyGxdqOBnfaks5s7_4ygaJpZM4FVGp6 .
maybe just add before_send hook as last point where request can be changed so by using one hook we can inject auth headers before send request ?
I was just digging into a similar topic in the last days: OpenID Connect, OAuth2, validation of JWT issued by Azure B2C in Python, I made a working setup for servers (will share my code when I have time to clean it up).
I think aiohttp.client and ClientSession should not support more options for auth, or support handling of specific response codes (like 401, 403); these classes must be abstracted from specific auth scenarios: they already have lots to do, just to be good HTTP clients and client sessions.
In fact, I think that including support for basic auth in the constructor of ClientSession was a mistake in the first place. There are just too many scenarios: should the client support automatic login using Client Credentials flow, by client id and secret? Or maybe automatic handling of refresh_tokens to fetch new access_tokens when using Authorization code grant?
I think these things should be placed in dedicated libraries, outside of aiohttp.
Perhaps. But it would be useful to provide a means of doing so. Do you agree?
I agree it would be useful, but not inside aiohttp. I think that handling specific scenarios should reside on a upper level, for example a class that internally uses an http client, and in dedicated libraries.
For example, if I had to support handling of refresh_tokens to obtain new access_tokens like I mentioned previously, I would do it in a specific class that uses an http client as one of its dependencies, and I wouldn't put it inside aiohttp, but in a dedicated library like aiohttp-oauth.
Agreed. However this would still require a refactor of the library's API to support something like ntlm auth where you will need to reuse connections.
Sent from Yahoo Mail on Android
On Fri, Oct 12, 2018 at 11:16 AM, Roberto Prevato[email protected] wrote:
I agree it would be useful, but not inside aiohttp. I think that handling specific scenarios should reside on a upper level, for example a class that internally uses an http client, and in dedicated libraries.
For example, if I had to support handling of refresh_tokens to obtain new access_tokens like I mentioned previously, I would do it in a specific class that uses an http client as one of its dependencies, and I wouldn't put it inside aiohttp, but in a dedicated library like aiohttp-oauth.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.
In this case yes, if you need a way to reuse connections and this is not possible, but still I think it should be abstracted from NTLM. I just took a look at the implementation in client.py and it is currently bound to BasicAuth class. I understand you would need something like middlewares, but for client.
Any plans to make this scenario easier to accommodate? I managed to get NTLM working, but it's not clear how to integrate it in a reliable way. My current thinking is to pass my own implementation of ClientResponse to ClientSession which delegates everything to the library implementation of ClientResponse with the exception of the start method, in which I could issue additional requests in case the request requires authentication. Is that something that can be recommended by the library developers?
@letrec any chance of you providing that code? I've been trying to do the same
@Mause, I'm in the middle of implementing the approach I outlined, but keep hitting all kinds of roadblocks starting from ClientResponse using not exported types in it's public interface and ClientSession directly accessing private fields of ClientResponse. Getting more and more pessimistic with every iteration.
Actually, I was wrong about the usage of private fields. Managed to get it working, but need some time to think on final design.
In NTLM one authenticates connections, not requests (https://blogs.technet.microsoft.com/mist/2018/02/14/windows-authentication-http-request-flow-in-iis/). So if a request fails because it wasn't authenticated we need to follow it up with negotiate and authenticate requests on the same connection as the initial request. If we just issues them on the same instance of ClientSession there's no guarantee that the connection will be the same. This behaviour can be achieved if session is configured to use a connector with its connection limit set to 1, but it's not going to scale well. It look like we need a special kind of connector which allocates connections from another connector and returns them to the parent connector only when it is closed. We could create such a connector and set its connection limit to 1 and use it to create sessions that need such behaviour. In this case the client code might look like this:
async def ntlm_get_text(connector: aiohttp.BaseConnector, url: str, domain: str, username: str, password: str) -> str:
async with ChildConnector(connector=connector) as child_connector:
async with aiohttp.ClientSession(connector=child_connector) as session:
async with session.get(url) as response:
# NTLM-specific status handling with required retries
return await response.text()
@asvetlov, does this approach seem feasible to you?
@asvetlov, you can ignore the NTLM part. The problem boils down to being able to create child connectors, which borrow a subset of connections from their parent connector. It might be useful for other purposes as well, like having different limits for different hosts, for instance. The issue is that create_child_connector has to be implemented in BaseConnector for all derived classes to benefit from it. I don't see how it can be implemented externally.
I still think that nested connectors drive to overcomplicated application design,
BaseConnector allows to set limit per host, but it's going to be the same limit for all hosts, if I understand it correctly. What I'm proposing is a generalization of this mechanism which could allow more fine-grained control over connections without introducing new abstractions. Limit per host could be accomplished in terms of this mechanism but with more flexibility. In my mind this idea is conceptually similar to context managers or trio cancel scopes / nurseries.
My current approach is to create a connector with limit set to one and a new session per each request that requires authentication, so the problem is kinda solved albeit not in the most performant way, but good enough for my purposes.
I likewise have a need for more control over authentication, and like others that have already posted, to support NTLM SSPI.
Basically, the use case at my organization is that we have a number of webservices running under IIS on Windows that require authentication. Our python clients are also running from Windows.
Currently, we're using requests and requests_negotiate_sspi so that we don't have to explicitly specify credentials.
To achieve parallelism currently, we're using multiprocessing, which has a fair amount of per request overhead. The overhead is compounded when the python install is located on a network share due to our corporate mandated virus scan interfering on every new subprocess launched (seriously, it's about 1-2 minutes before a new Python process does anything useful when it's on a network share at my company, and no, we can't disable it because of hypersensitivity over corporate information security at a financial company).
aiohttp looked very promising when I cut a branch of our internal client to use it (saw an immediate 28% reduction in wall-clock time) when running against a local dev build of our webservice that didn't require auth. But, I cannot run my new version in either our QA or PROD environments because they require authentication (all I get is 401 errors).
I'm looking at adding support for SSPI in our client, but it is certainly not trivial. I think it's complicated enough and needed by enough people that it's worth adding to either aiohttp or a submodule (and looking around the project, submodule probably makes more sense). I don't currently have enough knowledge of aiohttp to really know how/where to integrate such an add-in, or if it's even really currently possible.
But yes, to me (and apparently others), authentication support beyond basic auth is a must have.
