[BUG] Running python step fails when it tries to install the wrong dependency

[Stvad]

Describe the bug When running a python step that depends on py-august package that exports august package (and adding a dependency via magic comment). The executor still tries to install the august package which fails overall step because it depends on Rust (https://pypi.org/project/august/)

To Reproduce Steps to reproduce the behavior:

Run the following Python step

# pipedream add-package py-august
from august.api import Api
from august.authenticator import Authenticator, AuthenticationState


def handler(pd: "pipedream"):
  return {"test": True}

It'll fail with the error like:

Command failed: python3 -m pip install --target=/tmp/__pdg__/dist/python -r requirements.txt --upgrade WARNING: The directory '/home/sbx_user1051/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. error: subprocess-exited-with-error × Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [6 lines of output] Cargo, the Rust package manager, is not installed or is not on PATH. This package requires Rust and Cargo to compile extensions. Install it through the system's package manager or via https://rustup.rs/ Checking for Rust toolchain.... [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available. You should consider upgrading via the '/var/lang/bin/python3 -m pip install --upgrade pip' command. 

Expected behavior The environment does not try to install august package and only installs py-august (and doesn't crash as a consequence 😛)

[ghost]

@Stvad thanks for reporting the issue iam able to reproduce the issue

[elibenton]

Having the same issue with another python package. Eager to here how it's resolved. Thanks!

[ghost]

@elibenton what other python package you are facing issue with kinldy report here so it will be resolved

Having the same issue with another python package. Eager to here how it's resolved. Thanks!

[elibenton]

Thanks, Hassan! It's an old article scraping package called newspaper3k.

https://pypi.org/project/newspaper3k/

[stefanbohacek]

Also having this issue, with mastodon.py.

[dylburger]

@stefanbohacek We just corrected the issue. You should be able to from mastodon import Mastodon successfully now.

@elibenton We also addressed the issue with newspaper3k. Try from newspaper import Article again.

The root issue is the same in each case. The system we use to magically map import statements to PyPI packages works like this:

1. Scan code and collect imports
2. Check if a package with the import name exists in PyPI. If yes, write to the requirements.txt file.
3. (handle other edge cases)

In these two cases, there are technically packages for mastodon and newspaper. mastodon is unmaintained and doesn't properly package Python code, so fails on install. newspaper points to the Python 2 version of the package. So these packages were written to the requirements.txt file, but failed to install.

We added custom mappings to point these import names to the correct PyPI package names:

newspaper:newspaper3k
mastodon:Mastodon.py

If anyone else finds this issue and is having the same problem, please open up a new GitHub issue with the build error you're having and we'll take a look. We also plan to improve the mapping system, and will likely provide users the ability to provide a custom requirements file so you can override the magic imports if necessary.

@Stvad as you noted, your issue results from Rust not being part of our execution environment. I'm keeping this issue open so we can address that later.

[stefanbohacek]

Thank you @dylburger, this is now working for me!

Really like the "magical" import, by the way.

[Stvad]

@dylburger In my case that is the same problem as in the other 2 cases, Rust is pulled in by the incorrect package (august). the correct package (py-august) does not require Rust.

So the following mapping needs to exist as-well august:py-august though given that the august package also exists it's not clear that mapping is even the way to resolve this correctly.

I also don't think it's sustainable to ask you to add mapping for each package with naming mismatch - IMO it should be something that people are able to specify per-workflow

[markvh033]

Im glad this is being looked at! A custom requirements.txt will probably be a good idea.

I have an issue with an Azure package. Azure recently removed their standalone 'azure' package and created separate packages. I think pipreqs still thinks you need the 'azure' package.

Code:

Expected: only azure-storage-file-datalake is installed, and it doesn't try and install the 'azure' package, and thus doesn't fail.

Error:

Thanks!

[andrewjschuang]

@Stvad You should be able to import august from now on!