zenml
zenml copied to clipboard
zenml-io
Resolve Vertex Scheduler Character Limit Issue for Long Pipeline Names
Open Source Contributors Welcomed!
Please comment below if you would like to work on this issue!
Contact Details [Optional]
What happened?
A user encountered an issue where the Vertex scheduler in ZenML does not accept pipeline names longer than 64 characters. This limitation requires users to shorten their pipeline names to successfully schedule their pipelines on Vertex AI, which can be restrictive and inconvenient.
Task Description
Investigate and address the issue with the Vertex scheduler's character limit for pipeline names in ZenML. The goal is to either increase the character limit or implement a mechanism that automatically handles longer names without user intervention.
Expected Outcome
Users should be able to schedule pipelines on Vertex AI without the constraint of a 64-character limit on pipeline names. ZenML should provide a seamless experience, either by accommodating longer names or by intelligently managing them within the existing Vertex scheduler limitations. This fix will enhance user convenience and flexibility in naming pipelines.
Steps to Implement
- Reproduce the issue to understand the exact limitations and behavior of the Vertex scheduler with respect to pipeline names.
- Explore solutions such as increasing the character limit or automatically abbreviating/transforming long pipeline names while preserving their uniqueness and recognizability.
- Implement the chosen solution in a way that is transparent and intuitive for the user.
- Test the implementation with various pipeline name lengths to ensure compatibility and functionality.
- Update documentation to inform users about the handling of pipeline names in Vertex scheduler and any best practices or limitations.
Additional Context
This issue is crucial for users who require descriptive and potentially lengthy names for their pipelines, especially in complex or large-scale MLOps environments.
Code of Conduct
- [ ] I agree to follow this project's Code of Conduct
@christianversloot if I'm not completely mistaken, we had a conversation about this in Slack? Unfortunately the slack history hasn't been retained.
Correct, but that was for SageMaker orchestrator. There, I believe we limit name length to circumvent this error. It could perhaps be inspiration for a fix here :)
@christianversloot ah yes! https://github.com/zenml-io/zenml/pull/1505 was the PR.
@strickvl @christianversloot I would like to work on this... Please assign!
The bug has not been solved yet @Merthoshan. @ashutosh887 are you working on it, if not, can let @Merthoshan take a shot.
Where can we discuss if we're facing some issues understanding the bug? @strickvl I've worked it