prefect
prefect copied to clipboard
FileNotFoundError when building a Deployment with LocalFileSystem Block
First check
- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.
Bug summary
By default, without any additional parameters for a "deployment build", it defaults to a temporary location, using the process infrastructure block.
If a LocalFileSystem block is registered, and passed as a storage-block, the deployment build fails with
FileNotFoundError: [Errno 2] No such file or directory: ''
This occurs in both cases:
- When a path is specified, such as /tmp or my current working directory on my local system
- When no path is specified (the block creation lists path as 'Optional')
Reproduction
This is all completed on my local system.
#Create a new venv
python3 -m venv missing_dir_issue
#Activate
source missing_dir_issue/bin/activate
# Install prefect
pip3 install prefect
#Make folder for flows, and copy in healthcheck.py
mkdir missing_dir_issue/flows
cd missing_dir_issue/flows
#create healthcheck.py
cat <<EOF > healthcheck.py
import prefect
from prefect import task, flow
from prefect import get_run_logger
@task
def log_platform_info():
import platform
import sys
from prefect.orion.api.server import ORION_API_VERSION
logger = get_run_logger()
logger.info("Host's network name = %s", platform.node())
logger.info("Python version = %s", platform.python_version())
logger.info("Platform information (instance type) = %s ", platform.platform())
logger.info("OS/Arch = %s/%s", sys.platform, platform.machine())
logger.info("Prefect Version = %s 🚀", prefect.__version__)
logger.info("Prefect API Version = %s", ORION_API_VERSION)
@flow
def healthcheck():
log_platform_info()
if __name__ == "__main__":
healthcheck()
EOF
# Confirm version
prefect version
Version: 2.6.3
API version: 0.8.2
Python version: 3.9.12
Git commit: 9e7da96e
Built: Tue, Oct 18, 2022 1:55 PM
OS/Arch: darwin/x86_64
Profile: boyd
Server type: cloud
#Authenticate to cloud
prefect cloud login -k <>
#Create a local file system block ; two can be created, one with a path, and one left blank
prefect block create local-file-systemprefect block create local-file-system
Create a local-file-system block:
https://app.prefect.cloud/account/e<>/workspace/<>/blocks/catalog/local-file-system/create
# Build deployment, specifying the local filesystem block
prefect deployment build ./healthcheck.py:healthcheck -n dir_issue -sb local-file-system/test
Error
prefect deployment build ./healthcheck.py:healthcheck -n dir_issue -sb local-file-system/dir-fs-test
Found flow 'healthcheck'
Traceback (most recent call last):
File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 201, in coroutine_wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 152, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/Users/christopherboyd/opt/anaconda3/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/Users/christopherboyd/opt/anaconda3/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/prefect/cli/deployment.py", line 853, in build
deployment = await Deployment.build_from_flow(
File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/prefect/deployments.py", line 719, in build_from_flow
await deployment.upload_to_storage()
File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/prefect/deployments.py", line 572, in upload_to_storage
file_count = await self.storage.put_directory(
File "/Users/christopherboyd/virtualenvs/missing_dir_issue/lib/python3.9/site-packages/prefect/filesystems.py", line 187, in put_directory
shutil.copytree(
File "/Users/christopherboyd/opt/anaconda3/lib/python3.9/shutil.py", line 566, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
File "/Users/christopherboyd/opt/anaconda3/lib/python3.9/shutil.py", line 467, in _copytree
os.makedirs(dst, exist_ok=dirs_exist_ok)
File "/Users/christopherboyd/opt/anaconda3/lib/python3.9/os.py", line 225, in makedirs
mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''
Versions
prefect version
Version: 2.6.3
API version: 0.8.2
Python version: 3.9.12
Git commit: 9e7da96e
Built: Tue, Oct 18, 2022 1:55 PM
OS/Arch: darwin/x86_64
Profile: boyd
Server type: cloud
Additional context
As mentioned previously, this seems to occur in both cases - with a local path specified in the block, and when it is left blank.