Core/fix register remote

[novahow]

Tracking issue

https://github.com/flyteorg/flyte/issues/4936

Why are the changes needed?

According to https://github.com/flyteorg/flyte/issues/4936, some users encountered difficulties when trying to call register_workflow. While register_scripts can serve as an alternative, setting up the source_path and module_name can be pretty annoying. This change also enables execution with a relative path. For example:

file structure test_register.py

import typing
from flytekit import task, workflow
from flytekit.tools.script_mode import _find_project_root
from flytekit import ImageSpec, Resources, task, workflow
import os
import inspect
import pathlib
import pdb

@task
def say_hello(name: str) -> str:
    return f"hello {name}!"


@task
def greeting_length(greeting: str) -> int:
    return len(greeting)


@workflow
def wf(name: str = "union") -> typing.Tuple[str, int]:
    greeting = say_hello(name=name)
    greeting_len = greeting_length(greeting=greeting)
    return greeting, greeting_len



if __name__ == "__main__":
    from flytekit.remote import FlyteRemote
    from flytekit.configuration import Config, SerializationSettings, ImageConfig

    remote = FlyteRemote(
        config=Config.for_sandbox(),
        default_project="flytesnacks",
        default_domain="development"
    )


    print(wf)
    remote_wf = remote.register_workflow_script_mode(
        entity=wf,
    )

    wf_exe = remote.execute(entity=remote_wf, inputs={"name": "mike"})
    print(wf_exe.id)

You can run python test_register.py or python ../test_register.py under func_dir without worrying about the source_path.

What changes were proposed in this pull request?

Added an function register_workflow_script_mode, which basically leverages register_scripts, but automatically handles the source_path and module_name params. Also resolved the problem of calling register_script in the __main__ block when the registered workflow is in the same file as the __main__ block.

How was this patch tested?

1. unit test
2. remote integration test
3. local testing

Since we didn't export FLYTE_PYTHON_PACKAGE_ROOT and an __init__.py resides under our module register_test, it raises an error After export works fine here an __init__.py also resides under issue_register, without export would raise an error execute with relative path, again, an __init__.py resides under issue_register, without export would raise an error

Conclusion:

when current working dir is inside the module and you want to execute register_workflow_script_mode, please run export FLYTE_PYTHON_PACKAGE_ROOT=".". While executing in the parent dir of the module works fine.

Test Importing workflows

Results of python entrypoint.py

init.py under register_test, need export

entrypoint.py

from issue_register.test_register import wf as sub_wf
from flytekit import workflow
import pdb
import os


@workflow
def wf(name: str = "union") -> str:
    return sub_wf(name=name)[0]

if __name__ == "__main__":
    from flytekit.remote import FlyteRemote
    from flytekit.configuration import Config, SerializationSettings, ImageConfig, FastSerializationSettings
    import pathlib
    CONFIG = os.environ.get("FLYTECTL_CONFIG", str(pathlib.Path.home() / ".flyte" / "config-sandbox.yaml"))

    remote = FlyteRemote(
        config=Config.auto(config_file=CONFIG),
        default_project="flytesnacks",
        default_domain="development"
    )


    remote_sub_wf = remote.register_workflow_script_mode(
        entity=sub_wf,
    )
    remote_wf = remote.register_workflow_script_mode(
        entity=wf,
    )
    from issue_register.func_dir.test_register_in_func import register_wf
    func_remote_wf = register_wf(remote)
    remote.execute(entity=func_remote_wf, inputs={"name": "mike666"}, wait=True)

    subwf_exe = remote.execute(entity=remote_sub_wf, inputs={"name": "mike666"}, wait=True)
    wf_exe = remote.execute(entity=remote_wf, inputs={"name": "mike"}, wait=True)
    # pdb.set_trace()
    print("Func Remote", func_remote_wf.id)
    print("Remote SubWF", subwf_exe.id)
    print("Remote WF", wf_exe.id)

test_register_in_func.py

from issue_register.test_register import wf


def register_wf(remote):
    from flytekit.remote import FlyteRemote
    from flytekit.configuration import Config, SerializationSettings, ImageConfig, FastSerializationSettings
    import pathlib
    import os


    remote_wf = remote.register_workflow_script_mode(
        entity=wf,
        version="f0.0.1",
    )


    return remote_wf

Setup process

Screenshots

test_register.py

entrypoint.py

Check all the applicable boxes

• [ ] I updated the documentation accordingly.
• [x] All new and existing tests passed.
• [x] All commits are signed-off.

Related PRs

Docs link

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 60.00000% with 8 lines in your changes are missing coverage. Please review.

Project coverage is 83.52%. Comparing base (afa1fc7) to head (508c5af). Report is 10 commits behind head on master.

:exclamation: Current head 508c5af differs from pull request most recent head 6922340. Consider uploading reports for the commit 6922340 to get more accurate results

Files	Patch %	Lines
flytekit/remote/remote.py	53.84%	3 Missing and 3 partials :warning:
flytekit/core/tracker.py	71.42%	2 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2303      +/-   ##
==========================================
+ Coverage   75.79%   83.52%   +7.72%     
==========================================
  Files         181      324     +143     
  Lines       18275    24735    +6460     
  Branches     2567     3520     +953     
==========================================
+ Hits        13852    20660    +6808     
+ Misses       3823     3440     -383     
- Partials      600      635      +35     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.