unstructured
unstructured copied to clipboard
feat: improve load_requirements in setup.py
Description
This PR adds a number of enhancements around how we dynamically generate the requirements defined in the setup.py file. All logic around this was moved out into a setup_utils.py file to support unit tests around the logic:
load_requirements: Current approach to dynamically generate the requirements was causing them to maintain the newline character at the end. This was removed. Also added support for the-rcharacter in*.infiles which recursively pull in other.infiles which would currently be ignored. This isn't being used at the moment so it wasn't causing a problem, but this is future looking. It also used to support a list of filenames but that was never used, isolated to take in only a string filename of a Path object.- New methods added to generate the dictionaries fo requirements, one for connector requirements and another for doc type requirements.
- Safegaurds added throughout to raise an error if something incorrect was added. These include the file not being found or it not having the expected
.insuffix.
@qued, repo uploaded to test pypi here: unstructured/0.13.3.dev3/. This breaks through because it looks for all subsequent dependencies from the same test pypi index and they don't all exist there (i.e. langdetect). Need a way to tell pip to use test pypi index only for the unstructured package.
@qued @rbiseck3 - Are you still working this PR?