[BUG] flytekit 1.16.6 and flyteidl 1.16.1 are incompatible during registration

[rdeaton]

Flyte & Flytekit version

flytekit 1.16.6 flyteidl 1.16.1

Describe the bug

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/fn/lib/venv/lib/python3.12/site-packages/flytekit/__init__.py", line 119, in <module>
    from flytekit.core.array_node_map_task import map_task
  File "/fn/lib/venv/lib/python3.12/site-packages/flytekit/core/array_node_map_task.py", line 14, in <module>
    from flytekit.core.array_node import array_node
  File "/fn/lib/venv/lib/python3.12/site-packages/flytekit/core/array_node.py", line 12, in <module>
    from flytekit.core.launch_plan import LaunchPlan, ReferenceLaunchPlan
  File "/fn/lib/venv/lib/python3.12/site-packages/flytekit/core/launch_plan.py", line 17, in <module>
    from flytekit.models.concurrency import ConcurrencyPolicy
  File "/fn/lib/venv/lib/python3.12/site-packages/flytekit/models/concurrency.py", line 6, in <module>
    class ConcurrencyLimitBehavior(object):
  File "/fn/lib/venv/lib/python3.12/site-packages/flytekit/models/concurrency.py", line 7, in ConcurrencyLimitBehavior
    SKIP = _launch_plan_idl.CONCURRENCY_LIMIT_BEHAVIOR_SKIP
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'flyteidl.admin.launch_plan_pb2' has no attribute 'CONCURRENCY_LIMIT_BEHAVIOR_SKIP'

Expected behavior

...

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• [x] Yes

Have you read the Code of Conduct?

• [x] Yes

[welcome[bot]]

Thank you for opening your first issue here! 🛠

[runllm[bot]]

It seems likely that flytekit 1.16.6 and flyteidl 1.16.1 are incompatible because flytekit expects certain attributes (like CONCURRENCY_LIMIT_BEHAVIOR_SKIP) in flyteidl.admin.launch_plan_pb2 that are missing in flyteidl 1.16.1. This kind of AttributeError typically indicates a version mismatch between flytekit and flyteidl, where the Python SDK (flytekit) is ahead of the protocol buffer definitions (flyteidl) and relies on newer features not present in the older flyteidl version.

My understanding is that you should always use matching versions of flytekit and flyteidl to avoid such incompatibilities. For example, if you are using flytekit 1.16.6, you should also use flyteidl 1.16.6, as the code and protobuf definitions are tightly coupled and versioned together. This is supported by the test code in flytekit, which references these attributes and expects them to exist in the corresponding flyteidl version (test_concurrency.py).

Would you like more detail on how to resolve or avoid these version mismatches?

Sources:

• test_concurrency.py

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @runllm to ask follow-up questions. See runllm.com for more!

[rdeaton]

The LLM is on the right track... but the problem isn't on our end. There are no flyteidl releases past 1.16.1, and flytekit doesn't pin flyteidl to a compatible version.

flytekit 1.16.5 works okay with flyteidl 1.16.1 while we wait for a patch here.

[Sovietaced]

Support for the concurrency limits came in flytekit 1.16.6: https://github.com/flyteorg/flytekit/releases/tag/v1.16.6

If what you're saying is true I don't see how the following unit tests could have passed: https://github.com/flyteorg/flytekit/pull/3267/files#diff-c4cfe855bcb35511ef7e3bfde95a23a24f49465fcf8adb81280213e3e4ddfdd3R7

[Sovietaced]

I can verify that I was seeing the same error until I updated flyteidl to v1.16.1 so I have to believe this is a user error.

v1.15.3

/Users/jasonparraga/code/flytekit/.venv/bin/python /Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pycharm/_jb_pytest_runner.py --target test_concurrency.py::test_concurrency_limit_behavior 
Testing started at 7:34 PM ...
Launching pytest with arguments test_concurrency.py::test_concurrency_limit_behavior --no-header --no-summary -q in /Users/jasonparraga/code/flytekit/tests/flytekit/unit/models

ImportError while loading conftest '/Users/jasonparraga/code/flytekit/tests/flytekit/conftest.py'.
../../conftest.py:5: in <module>
    import flytekit.configuration.plugin
../../../../flytekit/__init__.py:119: in <module>
    from flytekit.core.array_node_map_task import map_task
../../../../flytekit/core/array_node_map_task.py:14: in <module>
    from flytekit.core.array_node import array_node
../../../../flytekit/core/array_node.py:12: in <module>
    from flytekit.core.launch_plan import LaunchPlan, ReferenceLaunchPlan
../../../../flytekit/core/launch_plan.py:17: in <module>
    from flytekit.models.concurrency import ConcurrencyPolicy
../../../../flytekit/models/concurrency.py:6: in <module>
    class ConcurrencyLimitBehavior(object):
../../../../flytekit/models/concurrency.py:7: in ConcurrencyLimitBehavior
    SKIP = _launch_plan_idl.CONCURRENCY_LIMIT_BEHAVIOR_SKIP
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   AttributeError: module 'flyteidl.admin.launch_plan_pb2' has no attribute 'CONCURRENCY_LIMIT_BEHAVIOR_SKIP'

Process finished with exit code 4

v1.16.1

/Users/jasonparraga/code/flytekit/.venv/bin/python /Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pycharm/_jb_pytest_runner.py --target test_concurrency.py::test_concurrency_limit_behavior 
Testing started at 7:32 PM ...
Launching pytest with arguments test_concurrency.py::test_concurrency_limit_behavior --no-header --no-summary -q in /Users/jasonparraga/code/flytekit/tests/flytekit/unit/models

============================= test session starts ==============================
collecting ... collected 1 item

test_concurrency.py::test_concurrency_limit_behavior PASSED              [100%]

========================= 1 passed, 1 warning in 0.01s =========================

Process finished with exit code 0