DAGWorks-Inc
Build with Local Function Directly
--- PR TEMPLATE INSTRUCTIONS (1) ---
Address this Issue #685
Changes
- Added a local builder helper.
- Used python inspect to find out parent module and included with in sys modules.
How I tested this
Unit Tests
Notes
Checklist
- [x] PR has an informative and human-readable title (this will be pulled into the release notes)
- [x] Changes are limited to a single goal (no scope creep)
- [x] Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
- [x] Any change in functionality is tested
- [x] New functions are documented (with a description, list of inputs, and expected output)
- [x] Placeholder code is flagged / future TODOs are captured in comments
- [x] Project documentation has been updated if adding/changing functionality.
@swapdewalkar can you motivate this with an example please? I'm interested, curious to hear where/when you'd use it.
@skrawcz I saw the requirement here to get all fns that are defined in the local module and insert them into a driver.
dr = driver.Builder()
.with_local_modules()
.build()
instead of
if __name__ == '__main__':
import __main__
dr = driver.Builder()
.with_modules(__main__)
.build()
print(dr.execute(["foo"]))
Let me know if I understood something different from ticket.
@swapdewalkar cool makes sense -- want to comment on that ticket so I can assign it to you?
Otherwise I think we just need a test to exercise this properly.
@swapdewalkar thanks for the contribution!
The Python import system behaves in strange ways at times, so it would be beneficial if you could add to the docstring of .with_local_modules() a snippet of what Python code should be in your .py file and how to execute the code.
For instance, the current approach with inspect.stack()[1][0] seems to work for python -m run but will fail for python run.py of the same file. In that case, it can lead to hard to debug IndexError: list index out of range
A potential solution would be
def with_local_modules(self) -> "Builder":
"""Adds the local modules to the modules list.
:return: self
"""
module = __import__("__main__")
self.modules.append(module)
return self
This will get the module "__main__" from sys.modules and should be directly equivalent to
if __name__ == "__main__":
import __main__
But I invite you to investigate and set the appropriate tests!
