hamilton
hamilton copied to clipboard
DAGWorks-Inc
Configurable Validators
This allows us to configure validation on the per-node level. Done at config-time, this sets us up to determine how to add a validator from passed in configuration.
[Short description explaining the high-level reason for the pull request]
Changes
How I tested this
Notes
Checklist
- [ ] PR has an informative and human-readable title (this will be pulled into the release notes)
- [ ] Changes are limited to a single goal (no scope creep)
- [ ] Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
- [ ] Any change in functionality is tested
- [ ] New functions are documented (with a description, list of inputs, and expected output)
- [ ] Placeholder code is flagged / future TODOs are captured in comments
- [ ] Project documentation has been updated if adding/changing functionality.
Thoughts:
- Do we want to enable a per-data-quality check? Currently this is per-node. I like it this way as the API is simpler, but it could get annoying if just one check is breaking.
- What if there's a node named 'global'? (it shouldn't be a function, and can't be a kwarg...). So I'm ok with there not being one)
- Need to add more tests
Re: (1) I think we can add an extra field later for this (or add a subfield for check name) but otherwise I'm pretty happy with this. The tricky part is that checks don't have identifiable names...
Ok, I'm going to think through this API this weekend -- feel more confident about the others so i might hold off on this for a bit