prefect icon indicating copy to clipboard operation
prefect copied to clipboard

Docs for result persistence location is out of date and confusing.

Open taylor-curran opened this issue 1 year ago • 3 comments

First check

  • [X] I added a descriptive title to this issue.
  • [X] I used GitHub search to find a similar request and didn't find it 😇

Describe the issue

Link to part of the docs that needs to be updated: https://github.com/PrefectHQ/prefect/blob/b3d98004a6748f07feb94a0c1899800804635969/docs/3.0rc/develop/results.mdx#result-storage-location

I have seen people be super confused that the block is defined in the decorator and also that with_options is used instead of the flow decorator.

Also S3 as a block will no longer be supported in favor of the s3 bucket block from the prefect-aws library.

new_flow = my_flow.with_options(result_storage=S3(bucket_path="my-bucket")) ^this is super confusing

from prefect import flow, task
from prefect.filesystems import LocalFileSystem, S3

@flow(persist_result=True)
def my_flow():
    my_task()  # This task will use the flow's result storage

@task(persist_result=True)
def my_task():
    ...

my_flow()  # The flow has no result storage configured and no parent, the local file system will be used.


# Reconfigure the flow to use a different storage type
new_flow = my_flow.with_options(result_storage=S3(bucket_path="my-bucket"))

new_flow()  # The flow and task within it will use S3 for result storage.

Describe the proposed change

Most users when they set up result persistence for the first time, they will be using a flow decorator.

First tell them to create the block, then have them reference the block from the flow decorator.

Use S3Bucket instead of S3 so we don't encourage them to adopt something that is not supported.

from prefect_aws.s3 import S3Bucket

@flow(persist_result=True, result_storage=S3Bucket.load("result-storage"))
def asyncio_gather_sub_deployments(
    sim_failure: SimulatedFailure = SimulatedFailure(), sleep_time_subflows: int = 0
):


Additional context

No response

taylor-curran avatar Jun 25 '24 19:06 taylor-curran

I'll take this up today or tomorrow 🫡

cicdw avatar Jun 25 '24 19:06 cicdw

Thanks! Adding info on this setting

ENV PREFECT_DEFAULT_RESULT_STORAGE_BLOCK

from #10925 could also be helpful.

I get this question from customers all the time:

Can I set global result location and forgo defining it again and again in each flow decorator?

taylor-curran avatar Jun 25 '24 19:06 taylor-curran

Awesome, will definitely cover that!

cicdw avatar Jun 25 '24 20:06 cicdw