Deployment updates

[cicdw]

Summary

This PR is the first in a series of changes that update deployments in a few different ways based on feedback; the current PR:

• removes the manifest.json files
• updates deployment schemas to include 3 new fields, and expands build CLI to support these fields
• stops saving local file system blocks automatically
• exposes a way to provide targeted infrastructure overrides via dot-delimited paths

And follow-up PRs will expand upon this with:

• Python-based deployment creation
• better Docker support
• and last but not least improved documentation around recipes and best practices

Steps Taken to QA Changes

• updated tests, wrote new tests
• tested using some QA projects
• @anna-geller is supporting

[anna-geller]

QA looks really good so far. I reported some feedback to Chris directly. Pasting some QA commands if anyone else wants to use them:

# QA of overrides - local storage + local process infra block
prefect deployment build flows/healthcheck.py:healthcheck --name process --tag dev -o deployments/process.yaml -ib process/dev -v my_github_sha
prefect deployment apply deployments/process.yaml
prefect deployment run healthcheck/process

prefect deployment build flows/healthcheck.py:healthcheck --name process2 --tag dev -o deployments/process2.yaml -ib process/dev --override env.PREFECT_LOGGING_LEVEL=INFO -v my_github_sha
prefect deployment apply deployments/process2.yaml
prefect deployment run healthcheck/process2
# validated log level changed due to overrides ✅ 


# Manifest + local FS block removal QA - local storage + local process explicit 
# validated that no block gets created for local FS, instead we now only get path and entrypoint ✅ 
prefect deployment build flows/healthcheck.py:healthcheck --name explicitprocess --tag dev -o deployments/explicitprocess.yaml -v my_github_sha --infra process
prefect deployment apply deployments/explicitprocess.yaml
prefect deployment run healthcheck/explicitprocess


# Manifest removal QA - local storage + local process implicit - validated no JSON manifest gets created ✅ 
prefect deployment build --help
prefect deployment build flows/healthcheck.py:healthcheck --name byejson --tag dev -o deployments/byejson.yaml -v my_github_sha
prefect deployment apply deployments/byejson.yaml
prefect deployment run healthcheck/byejson


# Creating s3/dev block, then QA with S3 + local process - S3 path gets uploaded/downloaded properly ✅ 
prefect deployment build flows/healthcheck.py:healthcheck --name s3 --tag dev -o deployments/s3.yaml -sb s3/dev -v my_github_sha
prefect deployment apply deployments/s3.yaml
prefect deployment run healthcheck/s3

# Creating gcs/dev block, then QA with GCS + local process
prefect deployment build flows/healthcheck.py:healthcheck --name gcs --tag dev -o deployments/gcs.yaml -sb gcs/dev -v my_github_sha
prefect deployment apply deployments/gcs.yaml
prefect deployment run healthcheck/gcs

# Creating azure/dev block, then QA with Azure + local process
prefect deployment build flows/healthcheck.py:healthcheck --name azure --tag dev -o deployments/azure.yaml -sb azure/dev -v my_github_sha
prefect deployment apply deployments/azure.yaml
prefect deployment run healthcheck/azure
# throws weird warnings, but works:  20:38:29.110 | WARNING | msrest.serialization - Ran into a deserialization error. Ignoring since this is failsafe deserialization


# used this to add s3fs dependency https://docs.prefect.io/concepts/filesystems/#filesystem-specific-package-dependencies-in-docker-images 
# implicit Docker infra with S3 block
prefect deployment build flows/healthcheck.py:healthcheck --name docker-implicit-block --tag dev -o deployments/docker_implicit_block.yaml --infra docker-container -sb s3/dev
prefect deployment apply deployments/docker_implicit_block.yaml
prefect deployment run healthcheck/docker-implicit-block

# explicit Docker infra with S3 block
prefect deployment build flows/healthcheck.py:healthcheck --name dockerdev --tag dev -sb s3/dev -ib docker-container/dev -o deployments/dockerdev.yaml
prefect deployment apply deployments/dockerdev.yaml
prefect deployment run healthcheck/dockerdev

I found some edge cases with:

• Docker volumes which I can't get to work yet (the intention was to mount a local directory with flows + provide a path and entrypoint on the YAML manifest, and this way, avoid having to build a custom image)
• custom ECR images - I opened a separate issue for this

But those can be addressed separately, so this passes QA and can be merged 👍