airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Attempt to update Connexion library to version 3

Open VladaZakharova opened this issue 1 year ago • 14 comments

@RobbeSneyders @vincbeck @potiuk Hi all! I have opened a PR in main repository so more people can be involved in this discussion :) based on the conversation in this issue https://github.com/apache/airflow/issues/35234, i am trying to update BaseAuthManager() class to use FlaskApp instead of FlaskApi as suggested here: https://github.com/apache/airflow/issues/35234#issuecomment-1793449425 The problem that i have faced is trying to update BaseAuthManager class will lead to errors on the users side who actually use this class with FlaskApi return type from method get_api_endpoints(). Another idea from the Robbe came to fix this problem by combining functionality from Connexion 2 and Connexion 3, but i am not sure how it will look like. Any ideas will be appreciated! Thank you :) P.S. I am not an expert in this specific part of Airflow, but will be happy to become one :D

Co-Authored: @RobbeSneyders


^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

VladaZakharova avatar Dec 04 '23 14:12 VladaZakharova

Thanks @VladaZakharova.

I believe we'll need to take a slightly different approach though. We only want to create a single FlaskApp and register Apis on them. I left some comments.

As mentioned, I started an effort to port to Connexion 3 a couple of weeks ago, but didn't have the time to complete it. I pushed my changes here so you can have a look.

Thank you for your suggestions! I also checked your PR you have mentioned and i can say we are almost there! :D I think the best approach here will be to let Robbe create PR in Community with his changes, since all the migration on Apache Airflow side was covered in his PR. I was also able to test changes with new Gunicorn command to use ASGI as was suggested earlier also by Robbe and i can see that Airflow is actually working in breeze, but when clicking something in the UI i can see this error:

webserver  | [2023-12-07 10:27:48 +0000] [1312734] [ERROR] Exception in ASGI application
webserver  | Traceback (most recent call last):
webserver  | File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 164, in __call__
webserver  | await self.app(scope, receive, _send)
webserver  | File "/usr/local/lib/python3.8/site-packages/connexion/middleware/exceptions.py", line 101, in __call__
webserver  | await super().__call__(scope, receive, send)
webserver  | File "/usr/local/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
webserver  | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
webserver  | File "/usr/local/lib/python3.8/site-packages/starlette/_exception_handler.py", line 63, in wrapped_app
webserver  | response = await handler(conn, exc)
webserver  | File "/usr/local/lib/python3.8/site-packages/connexion/middleware/exceptions.py", line 39, in wrapper
webserver  | response = await run_in_threadpool(handler, request, exc)
webserver  | File "/usr/local/lib/python3.8/site-packages/starlette/concurrency.py", line 35, in run_in_threadpool
webserver  | return await anyio.to_thread.run_sync(func, *args)
webserver  | File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 49, in run_sync
webserver  | return await get_async_backend().run_sync_in_worker_thread(
webserver  | File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 2103, in run_sync_in_worker_thread
webserver  | return await future
webserver  | File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 823, in run
webserver  | result = context.run(func, *args)
webserver  | File "/opt/airflow/airflow/www/extensions/init_views.py", line 229, in _handle_api_not_found
webserver  | return views.not_found(ex)
webserver  | File "/opt/airflow/airflow/www/views.py", line 632, in not_found
webserver  | render_template(
webserver  | File "/usr/local/lib/python3.8/site-packages/flask/templating.py", line 145, in render_template
webserver  | app = current_app._get_current_object()  # type: ignore[attr-defined]
webserver  | File "/usr/local/lib/python3.8/site-packages/werkzeug/local.py", line 508, in _get_current_object
webserver  | raise RuntimeError(unbound_message) from None
webserver  | RuntimeError: Working outside of application context.
webserver  | This typically means that you attempted to use functionality that needed
webserver  | the current application. To solve this, set up an application context
webserver  | with app.app_context(). See the documentation for more information.
webserver  | During handling of the above exception, another exception occurred:
webserver  | Traceback (most recent call last):
webserver  | File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
webserver  | result = await app(  # type: ignore[func-returns-value]
webserver  | File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
webserver  | return await self.app(scope, receive, send)
webserver  | File "/usr/local/lib/python3.8/site-packages/connexion/apps/abstract.py", line 284, in __call__
webserver  | return await self.middleware(scope, receive, send)
webserver  | File "/usr/local/lib/python3.8/site-packages/connexion/middleware/main.py", line 501, in __call__
webserver  | await self.app(scope, receive, send)
webserver  | File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
webserver  | await response(scope, receive, send)
webserver  | TypeError: 'coroutine' object is not callable
webserver  | 192.168.11.1:48504 - "GET /static/clusterActivity.js HTTP/1.1" 500

I think it is connected to the fact that we are trying now to run synchronous methods asynchronously, so we are failing :D Maybe guys you can suggest something regarding this? Should we rewrite all the calls that are synchronous or there are other approaches to solve this problem?

The final Gunicorn command i was using is:

run_args = [
            sys.executable,
            "-m",
            "gunicorn",
            "-k",
            "uvicorn.workers.UvicornWorker",
            "--workers",
            str(num_workers),
            "--timeout",
            str(worker_timeout),
            "--bind",
            args.hostname + ":" + str(args.port),
            "--name",
            "airflow-webserver",
            "--pid",
            pid_file,
            "--config",
            "python:airflow.www.gunicorn_config",
        ]

Thank you!

VladaZakharova avatar Dec 11 '23 09:12 VladaZakharova

Thank you everyone who is involved! :) I have updated the code and Gunicorn command as it was suggested by @RobbeSneyders in his PR (thank you, great job!) I have also added some custom Encoder to fix some issue that occurred after some changes made last 2 weeks. It seems now that webserver works as expected, however i am not sure about the correctness of the changes. Will be great if someone familiar will take a look :) The only problem that remains is warning about unclosed js and css files, like this: webserver | /usr/local/lib/python3.8/site-packages/a2wsgi/wsgi.py:246 ResourceWarning: unclosed file <_io.BufferedReader name='/opt/airflow/airflow/www/static/dist/grid.6154b81a58dba7c670ca.js'>

VladaZakharova avatar Dec 20 '23 18:12 VladaZakharova

Hello Team, I see several issues which we need to fix for finishing this PR.

  1. Fix unit tests which use app. After connexion update create_app function returns connexion_app instead of flask_app, but unit tests in most cases expect flask_app. In this case we need to update all our unit tests (we have ~1000 tests). List of errors which I found:

    • for test_client() we should use connexion_app not flask_app. I fixed this error for most tests in the last commit.
    • TypeError: get() got an unexpected keyword argument 'environ_overrides' this error is the most frequent now. For fixing it we need to find the equivalent for this variable environ_overrides in connexion's test_client(). I didn't find any equivalent in Connexion and Starlette docs. @RobbeSneyders could you help us with that?
    • AttributeError: 'Response' object has no attribute 'data' I think this happens because Response object from connexion is not the same then from flask.

    Also we have a big amount of different errors which do not have a similar pattern.

  2. Update init_api_auth_provider function. In get_api_endpoints function we use connexion_app.add_api and expect FlaskApi as a return object, but add_api function always return None. Previously we use FlaskApi object for taking blueprints and then extend our base_paths here: https://github.com/apache/airflow/pull/36052/files#diff-a1cf5a1eea3231a292d789d82df7943bfe8fa89ca955404b2f9cdaa25e08fa80R309 @vincbeck is it important to save this logic for fab_auth_manager or is it enough to register blueprints under connexion_app.add_api functionality?

MaksYermak avatar Jan 10 '24 14:01 MaksYermak

Update init_api_auth_provider function. In get_api_endpoints function we use connexion_app.add_api and expect FlaskApi as a return object, but add_api function always return None. Previously we use FlaskApi object for taking blueprints and then extend our base_paths here: https://github.com/apache/airflow/pull/36052/files#diff-a1cf5a1eea3231a292d789d82df7943bfe8fa89ca955404b2f9cdaa25e08fa80R309 @vincbeck is it important to save this logic for fab_auth_manager or is it enough to register blueprints under connexion_app.add_api functionality?

I am not sure this is relevant anymore. The signature of get_api_endpoints changed. See here. It should simplify the problem.

vincbeck avatar Jan 10 '24 16:01 vincbeck

Update init_api_auth_provider function. In get_api_endpoints function we use connexion_app.add_api and expect FlaskApi as a return object, but add_api function always return None. Previously we use FlaskApi object for taking blueprints and then extend our base_paths here: https://github.com/apache/airflow/pull/36052/files#diff-a1cf5a1eea3231a292d789d82df7943bfe8fa89ca955404b2f9cdaa25e08fa80R309 @vincbeck is it important to save this logic for fab_auth_manager or is it enough to register blueprints under connexion_app.add_api functionality?

I am not sure this is relevant anymore. The signature of get_api_endpoints changed. See here. It should simplify the problem.

Hi @vincbeck ! I think the question here was about the line where we are appending base_paths variable with new registered blueprint. With the new version of Connexion the way of registering blueprints will be changed to register in connexion_app. So the method get_api_endpoints() with new version of Connexion will return None instead of blueprint object. So, if we can't return blueprint from the get_api_endpoints() method, the base_paths variable couldn't be extended. Should we consider still updating base_paths with blueprint object? Is this variable used later somewhere? The if statement here shows that we still can continue without adding blueprint to this list of base_paths. If we need this variable later, is there other way to add blueprint not using value returned by get_api_endpoints() since now it will return None?

def init_api_auth_provider(connexion_app: connexion.FlaskApp):
    """Initialize the API offered by the auth manager."""
    auth_mgr = get_auth_manager()
    blueprint = auth_mgr.get_api_endpoints(connexion_app)
    if blueprint:
        base_paths.append(blueprint.url_prefix if blueprint.url_prefix else "")
        flask_app = connexion_app.app
        flask_app.extensions["csrf"].exempt(blueprint)

VladaZakharova avatar Jan 18 '24 14:01 VladaZakharova

@RobbeSneyders , hope you can help :)

TypeError: get() got an unexpected keyword argument 'environ_overrides' this error is the most frequent now. For fixing it we need to find the equivalent for this variable environ_overrides in connexion's test_client(). I didn't find any equivalent in Connexion and Starlette docs. @RobbeSneyders could you help us with that?

I think we also need some help in this one question, since it requires some understanding of Connexion library in general. For tests we also need ability to override variables, what was done in previous version of Connexion using environ_overrides variable. For now, using old variable we get error TypeError: get() got an unexpected keyword argument 'environ_overrides'. We didn't find anything about replacement for this variable in the documentation for new version, so maybe you can help us with this?

VladaZakharova avatar Jan 18 '24 14:01 VladaZakharova

Update init_api_auth_provider function. In get_api_endpoints function we use connexion_app.add_api and expect FlaskApi as a return object, but add_api function always return None. Previously we use FlaskApi object for taking blueprints and then extend our base_paths here: https://github.com/apache/airflow/pull/36052/files#diff-a1cf5a1eea3231a292d789d82df7943bfe8fa89ca955404b2f9cdaa25e08fa80R309 @vincbeck is it important to save this logic for fab_auth_manager or is it enough to register blueprints under connexion_app.add_api functionality?

I am not sure this is relevant anymore. The signature of get_api_endpoints changed. See here. It should simplify the problem.

Hi @vincbeck ! I think the question here was about the line where we are appending base_paths variable with new registered blueprint. With the new version of Connexion the way of registering blueprints will be changed to register in connexion_app. So the method get_api_endpoints() with new version of Connexion will return None instead of blueprint object. So, if we can't return blueprint from the get_api_endpoints() method, the base_paths variable couldn't be extended. Should we consider still updating base_paths with blueprint object? Is this variable used later somewhere? The if statement here shows that we still can continue without adding blueprint to this list of base_paths. If we need this variable later, is there other way to add blueprint not using value returned by get_api_endpoints() since now it will return None?

def init_api_auth_provider(connexion_app: connexion.FlaskApp):
    """Initialize the API offered by the auth manager."""
    auth_mgr = get_auth_manager()
    blueprint = auth_mgr.get_api_endpoints(connexion_app)
    if blueprint:
        base_paths.append(blueprint.url_prefix if blueprint.url_prefix else "")
        flask_app = connexion_app.app
        flask_app.extensions["csrf"].exempt(blueprint)

I think the way we should do it is really close to what you have now with few differences:

  • Rename get_api_endpoints to set_api_endpoints. The return type should be updated to None. Documentation should be updated as well to something like "Set API endpoint(s) definition for the auth manager.". This is a breaking change but nobody uses this interface yet, so it is a good time to do it.
  • This piece of code flask_app.extensions["csrf"].exempt(blueprint) should be moved in the set_api_endpoints method using appbuilder.app.extensions["csrf"].exempt(api.blueprint)

vincbeck avatar Jan 18 '24 16:01 vincbeck

You will also have to update the method in AwsAuthManager, but that should be straightforward

vincbeck avatar Jan 18 '24 16:01 vincbeck

I rebased this change with main here: https://github.com/lewijw/airflow/tree/connexion-update-version

After building and deploying an image into k8s, the webserver is not rendering well, and I am seeing this in the webserver log:

[2024-01-31 00:44:10 +0000] [14] [ERROR] Exception in ASGI application Traceback (most recent call last): File "/home/sas/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 164, in call await self.app(scope, receive, _send) File "/home/sas/.local/lib/python3.8/site-packages/connexion/middleware/exceptions.py", line 101, in call await super().call(scope, receive, send) File "/home/sas/.local/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 62, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/home/sas/.local/lib/python3.8/site-packages/starlette/_exception_handler.py", line 63, in wrapped_app response = await handler(conn, exc) File "/home/sas/.local/lib/python3.8/site-packages/connexion/middleware/exceptions.py", line 39, in wrapper response = await run_in_threadpool(handler, request, exc) File "/home/sas/.local/lib/python3.8/site-packages/starlette/concurrency.py", line 35, in run_in_threadpool return await anyio.to_thread.run_sync(func, *args) File "/home/sas/.local/lib/python3.8/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/home/sas/.local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 2134, in run_sync_in_worke r_thread return await future File "/home/sas/.local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 851, in run result = context.run(func, *args) File "/opt/airflow/airflow/www/extensions/init_views.py", line 229, in _handle_api_not_found return views.not_found(ex) File "/opt/airflow/airflow/www/views.py", line 631, in not_found render_template( File "/home/sas/.local/lib/python3.8/site-packages/flask/templating.py", line 145, in render_template app = current_app._get_current_object() # type: ignore[attr-defined] File "/home/sas/.local/lib/python3.8/site-packages/werkzeug/local.py", line 508, in _get_current_object raise RuntimeError(unbound_message) from None RuntimeError: Working outside of application context.

This typically means that you attempted to use functionality that needed the current application. To solve this, set up an application context with app.app_context(). See the documentation for more information.

Is anyone else seeing this? Let me know if you have any ideas as to how to fix it. Thanks.

lewijw avatar Jan 31 '24 01:01 lewijw

FYI. I was able to get past the prior issue by adding a "with":

def _handle_api_not_found(
    request: ConnexionRequest, ex: starlette.exceptions.HTTPException
) -> ConnexionResponse:
    with connexion_app.app.app_context():
        if any([request.url.path.startswith(p) for p in base_paths]):
            # 404 errors are never handled on the blueprint level
            # unless raised from a view func so actual 404 errors,
            # i.e. "no route for it" defined, need to be handled
            # here on the application level
            return _handle_http_exception(ex)
        else:
            return views.not_found(ex)

I expect something like it needs to be done with the other handlers.

After getting past this problem, I am running into another error message:

RuntimeError: Unable to build URLs outside an active request without 'SERVER_NAME' configured. Also configure 'APPLICATION_ROOT' and 'PREFERRED_URL_SCHEME' as needed.

So, it looks like something is not getting initialized.

lewijw avatar Feb 01 '24 14:02 lewijw

Hey @potiuk @vincbeck ! The latest update on this issue and the list of the errors we need to fix to be able to upgrade Connexion:

  1. Based on this discussion:

Update init_api_auth_provider function. In get_api_endpoints function we use connexion_app.add_api and expect FlaskApi as a return object, but add_api function always return None. Previously we use FlaskApi object for taking blueprints and then extend our base_paths here: https://github.com/apache/airflow/pull/36052/files#diff-a1cf5a1eea3231a292d789d82df7943bfe8fa89ca955404b2f9cdaa25e08fa80R309 @vincbeck is it important to save this logic for fab_auth_manager or is it enough to register blueprints under connexion_app.add_api functionality?

I am not sure this is relevant anymore. The signature of get_api_endpoints changed. See here. It should simplify the problem.

Hi @vincbeck ! I think the question here was about the line where we are appending base_paths variable with new registered blueprint. With the new version of Connexion the way of registering blueprints will be changed to register in connexion_app. So the method get_api_endpoints() with new version of Connexion will return None instead of blueprint object. So, if we can't return blueprint from the get_api_endpoints() method, the base_paths variable couldn't be extended. Should we consider still updating base_paths with blueprint object? Is this variable used later somewhere? The if statement here shows that we still can continue without adding blueprint to this list of base_paths. If we need this variable later, is there other way to add blueprint not using value returned by get_api_endpoints() since now it will return None?

def init_api_auth_provider(connexion_app: connexion.FlaskApp):
    """Initialize the API offered by the auth manager."""
    auth_mgr = get_auth_manager()
    blueprint = auth_mgr.get_api_endpoints(connexion_app)
    if blueprint:
        base_paths.append(blueprint.url_prefix if blueprint.url_prefix else "")
        flask_app = connexion_app.app
        flask_app.extensions["csrf"].exempt(blueprint)

I think the way we should do it is really close to what you have now with few differences:

  • Rename get_api_endpoints to set_api_endpoints. The return type should be updated to None. Documentation should be updated as well to something like "Set API endpoint(s) definition for the auth manager.". This is a breaking change but nobody uses this interface yet, so it is a good time to do it.
  • This piece of code flask_app.extensions["csrf"].exempt(blueprint) should be moved in the set_api_endpoints method using appbuilder.app.extensions["csrf"].exempt(api.blueprint)

Blueprint is now can't be retrieved from connexion app and there will be always None. Find a way to add csrf extension to a newly created blueprint using connexion: to retrieve blueprint object from connexion_app variable to save the current logic (flask_app.extensions["csrf"].exempt(blueprint)) or find a way to add this extension on connexion level(check the documentation for available options).

  1. Find a way to replace legacy environ_overrides method from old version of Connexion to fix the problem in tests TypeError: get() got an unexpected keyword argument 'environ_overrides'. Contact @RobbeSneyders for some help. Currently we didn't find anything in their documentation regarding new way how to override env variables.

  2. Fix unit tests which use app.

After connexion update create_app function returns connexion_app instead of flask_app, but unit tests in most cases expect flask_app. In this case we need to update all our unit tests (we have ~1000 tests). As a reference of existing errors please check the logs from the last build here:

  • In Providers-AWS,Google part fix problem with legacy environ_overrides as mentioned in step#2. https://github.com/apache/airflow/actions/runs/7462817178/job/20307226379#step:6:3267
  • In Core part there is a failing test related to incorrect encoding in the current implementation (check method here that can be related to the error here, maybe we can change the return value from flask to connexion_app. Needs some investigation. https://github.com/apache/airflow/actions/runs/7462817178/job/20307226379#step:6:5821
  • In WWW part fix failing tests with error AttributeError: 'FlaskApp' object has no attribute 'test_request_context'. Please, check the documentation here https://github.com/apache/airflow/actions/runs/7462817178/job/20307226379#step:6:6496
  • In CLI part fix asserts that are related to new way of calling Gunicorn command: https://github.com/apache/airflow/actions/runs/7462817178/job/20307226379#step:6:10464
  • In Other part fix errors connected to the incorrect header that is generated. This step requires some additional investigation, since the problem could be not connected to incorrect test, but incorrect setting of headers. https://github.com/apache/airflow/actions/runs/7462817178/job/20307226379#step:6:11744
  • In Providers-AWS part fix the problem with AttributeError: 'FlaskApp' object has no attribute 'test_request_context' error. Please, check the documentation here https://github.com/apache/airflow/actions/runs/7462817178/job/20307226379#step:6:20406
  • In Providers-Google part fix the problem with AttributeError: 'Response' object has no attribute 'data' that occurres because Response object from connexion is not the same then from flask. https://github.com/apache/airflow/actions/runs/7462817178/job/20307226379#step:6:22081

VladaZakharova avatar Feb 06 '24 16:02 VladaZakharova

Hi @vincbeck @VladaZakharova , I started working on the bugs Vlada listed above.

For the first bug, I just updated codes as Vinc's commented. Can you guide me on how to test it. I'm new to this repository so any guidance for this issue would be helpful to me.

Here is my PR draft https://github.com/apache/airflow/pull/37555

Satoshi-Sh avatar Feb 20 '24 00:02 Satoshi-Sh

Sorry for the late response, let me try to give some pointers for the open issues.

  1. Based on this discussion: Blueprint is now can't be retrieved from connexion app and there will be always None. Find a way to add csrf extension to a newly created blueprint using connexion: to retrieve blueprint object from connexion_app variable to save the current logic (flask_app.extensions["csrf"].exempt(blueprint)) or find a way to add this extension on connexion level(check the documentation for available options).

The most future-proof solution would be to move the csfr protection to a middleware such as this one. The skif_if_scope parameter seems to be a replacement for the exempt functionality you're using.

  1. Find a way to replace legacy environ_overrides method from old version of Connexion to fix the problem in tests TypeError: get() got an unexpected keyword argument 'environ_overrides'. Contact @RobbeSneyders for some help. Currently we didn't find anything in their documentation regarding new way how to override env variables.

There's no one-on-one replacement for this as far as I can tell. This is also not something Connexion can easily offer, since we just leverage the starlette test client. The easiest approach I can think of, is to add a middleware instead.

class RemoteUserMiddleware:

    def __init__(self, asgi_app: Callable, remote_user: str) -> None:
        self.asgi_app = asgi_app
        self.remote_user = remote_user

    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
        scope["REMOTE_USER"] = self.remote_user
        await self.asgi_app(scope, receive, send)

You could use this across tests like so:

app.add_middleware(RemoteUserMiddleware)
with app.test_client():
    ...

Note that:

  • This sets the REMOTE_USER on a test client level instead of a request level though.
  • You might have to update how this variable is accessed as well. request.scope["REMOTE_USER"] should work.
  1. Fix unit tests which use app.

After connexion update create_app function returns connexion_app instead of flask_app, but unit tests in most cases expect flask_app. In this case we need to update all our unit tests (we have ~1000 tests). As a reference of existing errors please check the logs from the last build here:

  • In Providers-AWS,Google part fix problem with legacy environ_overrides as mentioned in step#2. https://github.com/apache/airflow/actions/runs/7462817178/job/20307226379#step:6:3267

See above

  • In Core part there is a failing test related to incorrect encoding in the current implementation (check method here that can be related to the error here, maybe we can change the return value from flask to connexion_app. Needs some investigation. https://github.com/apache/airflow/actions/runs/7462817178/job/20307226379#step:6:5821

I think this is due to a change in Flask and a non-issue: https://github.com/pallets/flask/issues/5286

  • In WWW part fix failing tests with error AttributeError: 'FlaskApp' object has no attribute 'test_request_context'. Please, check the documentation here https://github.com/apache/airflow/actions/runs/7462817178/job/20307226379#step:6:6496

Two possibilities here:

  • Use the underlying Flask app: app.app.test_request_context()
  • Switch to the Connexion test context in the reference docs, but you will also need to change the context being used in the tested code.
  • In CLI part fix asserts that are related to new way of calling Gunicorn command: https://github.com/apache/airflow/actions/runs/7462817178/job/20307226379#step:6:10464

Looks like this just needs an update to the test code to match the renamed arguments.

  • In Other part fix errors connected to the incorrect header that is generated. This step requires some additional investigation, since the problem could be not connected to incorrect test, but incorrect setting of headers. https://github.com/apache/airflow/actions/runs/7462817178/job/20307226379#step:6:11744

I don't see how the linked issue is related to headers, but maybe I'm missing some context.

The error I see is

AttributeError: 'FlaskApp' object has no attribute 'config'

Which is because you're treating the Connexion App like a Flask App. Changing app.config to app.app.config should give you what you want.

  • In Providers-AWS part fix the problem with AttributeError: 'FlaskApp' object has no attribute 'test_request_context' error. Please, check the documentation here https://github.com/apache/airflow/actions/runs/7462817178/job/20307226379#step:6:20406

See above.

  • In Providers-Google part fix the problem with AttributeError: 'Response' object has no attribute 'data' that occurres because Response object from connexion is not the same then from flask. https://github.com/apache/airflow/actions/runs/7462817178/job/20307226379#step:6:22081

The returned response provides a lot of methods to access the data. The one on one translation is probably .content, but if you're decoding anyway, you could use .text instead.

Hope this helps!

RobbeSneyders avatar Feb 22 '24 00:02 RobbeSneyders

@RobbeSneyders Thanks for the pointers. I will have a look at it and ask you if I have a question.

Satoshi-Sh avatar Feb 22 '24 00:02 Satoshi-Sh

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Apr 08 '24 00:04 github-actions[bot]