prefect icon indicating copy to clipboard operation
prefect copied to clipboard

FileNotFoundError when building a Deployment with LocalFileSystem Block

Open prefectcboyd opened this issue 2 years ago • 0 comments

First check

  • [X] I added a descriptive title to this issue.
  • [X] I used the GitHub search to find a similar issue and didn't find it.
  • [X] I searched the Prefect documentation for this issue.
  • [X] I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

By default, without any additional parameters for a "deployment build", it defaults to a temporary location, using the process infrastructure block.

If a LocalFileSystem block is registered, and passed as a storage-block, the deployment build fails with

FileNotFoundError: [Errno 2] No such file or directory: ''

This occurs in both cases:

  1. When a path is specified, such as /tmp or my current working directory on my local system
  2. When no path is specified (the block creation lists path as 'Optional')

Reproduction

This is all completed on my local system.


#Create a new venv
python3 -m venv missing_dir_issue

#Activate
source missing_dir_issue/bin/activate

# Install prefect
pip3 install prefect

#Make folder for flows, and copy in healthcheck.py
mkdir missing_dir_issue/flows
cd missing_dir_issue/flows

#create healthcheck.py
cat <<EOF > healthcheck.py
import prefect
from prefect import task, flow
from prefect import get_run_logger

@task
def log_platform_info():
    import platform
    import sys
    from prefect.orion.api.server import ORION_API_VERSION
    logger = get_run_logger()
    logger.info("Host's network name = %s", platform.node())
    logger.info("Python version = %s", platform.python_version())
    logger.info("Platform information (instance type) = %s ", platform.platform())
    logger.info("OS/Arch = %s/%s", sys.platform, platform.machine())
    logger.info("Prefect Version = %s 🚀", prefect.__version__)
    logger.info("Prefect API Version = %s", ORION_API_VERSION)
@flow
def healthcheck():
    log_platform_info()
if __name__ == "__main__":
    healthcheck()
EOF


# Confirm version 

prefect version
Version:             2.6.3
API version:         0.8.2
Python version:      3.9.12
Git commit:          9e7da96e
Built:               Tue, Oct 18, 2022 1:55 PM
OS/Arch:             darwin/x86_64
Profile:             boyd
Server type:         cloud

#Authenticate to cloud 
prefect cloud login -k <>

#Create a local file system block ; two can be created, one with a path, and one left blank
prefect block create local-file-systemprefect block create local-file-system
Create a local-file-system block:
https://app.prefect.cloud/account/e<>/workspace/<>/blocks/catalog/local-file-system/create


# Build deployment, specifying the local filesystem block
prefect deployment build ./healthcheck.py:healthcheck -n dir_issue -sb local-file-system/test

Error

prefect deployment build ./healthcheck.py:healthcheck -n dir_issue  -sb local-file-system/dir-fs-test
Found flow 'healthcheck'
Traceback (most recent call last):
  File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 201, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 152, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/Users/christopherboyd/opt/anaconda3/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/christopherboyd/opt/anaconda3/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/prefect/cli/deployment.py", line 853, in build
    deployment = await Deployment.build_from_flow(
  File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/prefect/deployments.py", line 719, in build_from_flow
    await deployment.upload_to_storage()
  File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/prefect/deployments.py", line 572, in upload_to_storage
    file_count = await self.storage.put_directory(
  File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/prefect/filesystems.py", line 187, in put_directory
    shutil.copytree(
  File "/Users/christopherboyd/opt/anaconda3/lib/python3.9/shutil.py", line 566, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/Users/christopherboyd/opt/anaconda3/lib/python3.9/shutil.py", line 467, in _copytree
    os.makedirs(dst, exist_ok=dirs_exist_ok)
  File "/Users/christopherboyd/opt/anaconda3/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''

Versions

prefect version
Version:             2.6.3
API version:         0.8.2
Python version:      3.9.12
Git commit:          9e7da96e
Built:               Tue, Oct 18, 2022 1:55 PM
OS/Arch:             darwin/x86_64
Profile:             boyd
Server type:         cloud

Additional context

As mentioned previously, this seems to occur in both cases - with a local path specified in the block, and when it is left blank.

prefectcboyd avatar Oct 25 '22 17:10 prefectcboyd