Request for Enabling `include_local_workdir` Parameter Setting with `@step` Decorator in Local Mode

[grmn621]

Describe the feature you'd like I would like to request the addition of a parameter for setting include_local_workdir when using the @step decorator in pipeline's local mode. Currently, there is no way to set this parameter in local mode, which is quite inconvenient. While in non-local mode, it's possible to configure this through the session's SM config, and the @remote decorator has its own set of arguments, the absence of a similar feature for the @step decorator in local mode is a significant limitation.

How would this feature be used? Please describe. This feature would greatly enhance the usability of the @step decorator in local development scenarios by allowing developers to include their local working directory in the step's execution context. For example, if a developer is testing a pipeline locally and needs to include files from their local directory, they could do so easily by setting the include_local_workdir parameter. This would streamline the development and testing of pipelines, making the process more efficient and less error-prone.

Describe alternatives you've considered An alternative would be to manually copy necessary files into the main file for each execution, but this is cumbersome and prone to errors. Another possibility is to use remote execution for all testing, but this can be slower and more costly, especially for quick iteration cycles.

Additional context The @step decorator is an excellent feature, and enhancing its functionality for local development would make it even more valuable. The ability to include the local working directory seamlessly in the execution context would be a significant improvement for developers working with pipelines in a local environment. I have researched this thoroughly, but please let me know if there are any differing opinions or existing solutions I may have overlooked.

[nmadan]

Hi @grmn621,

Thanks for reaching out. Are you using a LocalPipelineSession? If yes, I believe this class also loads the sagemaker config defaults.

You should be able to override the config file name environment variable like this:

import os

# Set path to config file
os.environ["SAGEMAKER_USER_CONFIG_OVERRIDE"] = os.getcwd()

Your config file should look something like this- https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-pipelines/step-decorator/quick-start/notebooks/config.yaml

[qidewenwhen]

Hoping the reply from nmadan@ helps. Closing the issue as we have not heard back in past 3 weeks. Feel free to reopen if you have further questions. Thanks!