Using the official PyPi repo

[bostone]

I'm trying to define dependency that is not in the demo repo pyGradlePyPi(). I can't find anywhere in the documentation how to reference the official pypi.python.org repo and also if it's possible to reference Artifactory-hosted PyPi repo? Can anyone provide a hint?

[bostone]

Anyone? I'm about to give up on pygradle altogether

[sholsapp]

From https://github.com/linkedin/pygradle#pypi-artifacts...

... we cannot use pypi directly.

... because PyPI currently doesn't host the Ivy metadata that Gradle needs to do proper dependency resolution.

From the same document, there is a link to read more about how to use the import tool that we provide to generate that Ivy metadata. You can configure your own Artifactory-hosted PyPI and use the import tool to generate and publish the Ivy metadata to your own Artifactory.

If you need more help learning how to use Gradle to configure a remote repository like Artifactory, we can dig up an example for you.

[bostone]

Thanks, I was actually reading those realizing that you guys should probably emphasize it better when I got your message. We do have internal Artifactory-pro that can host both Ivy and PyPi repos. I'm just not clear how to proceed? Right now I only need PyYAML library which is not in our Artifactory. I think example of setting up Artifactory-hosted PyPi would be extremely nice

-Bo

[lorencarvalho]

@sholsapp we should create a few examples of how to set up a local pypi mirror (a.k.a. vendoring to disk (a la https://github.com/linkedin/pygradle/blob/master/examples/iris-classification/build.gradle#L30-L36) and also some instructions on using artifactory and/or bintray.

[bostone]

I'm also thinking of adding Java nature to my gradle project to obtain pivy-importer dynamically and adding task that would run a jar and create a local repo. People who would use my code would be highly annoyed with idea of getting the jar and creating the repo manually. Any thoughts?

-Bo

[sholsapp]

Okay, will do, that's easy to document. We also could do a quick walk through of how we setup our Artifactory and run our import tool to get someone up and running quickly.

[lorencarvalho]

I think it would be valuable to provide some automation (either a script or perhaps a gradle task) around creating per-project bintray repos! That way someone could wire in the repo with their project and not have to worry about bootstrapping consumers.

[bostone]

BTW - if you want to get credit and help more people I have a Stackoverflow question for this

[bostone]

Any progress on this? Meanwhile I managed to use python-sdist to generate an egg (a wheel?). Can you guys tell me how to install it into the Artifactory-hosted PyPi. The instructions I have tell me to do python setup.py sdist upload -r local but when I execute that from the top of my gradle project I get UNKNOWN-0.0.0.tar.gz created (and pushed) which actually does include all the correct files but the configuration is hopelessly messed up

[nimish]

This is basically a showstopper -- is there a way to dynamically generate the ivy files on each build? I would really not want to run my own repo...

[bostone]

@nimish Unless you decide to code it yourself I understand that there's nothing like that at this point (at least publicly). For my much simpler case I just ended up calling Python commands from Gradle. Something like this:

/** Creates the distribution and installs wheel into Artifactory's PyPI */
task wheel(dependsOn: test) {
    doLast {
        println "Copying Artifactory PiPY config file"
        copy {
            duplicatesStrategy = DuplicatesStrategy.WARN
            from resDir
            into userHome
            include ".pypirc"
        }
        println "Building the distro and deploying to artifactory"
        exec {
            workingDir src
            commandLine 'python', 'setup.py', 'bdist_wheel', 'upload', '-r', 'local'
        }
    }
}

[nimish]

@bostone I'm more interested in resolving PyPI artifacts than publishing mine to an internal PyPI repo and it looks like the gradle folks need help to extend the repo handling: https://github.com/gradle/gradle/issues/1400

This, ideally, would obviate the need to generate ivy metadata...

[zvezdan]

@nimish I am not sure this would obviate the need for Ivy metadata. We use Gradle for dependency resolution (its strength) and thus avoid version conflicts that are typical with python build tools. This is where Ivy helps. Once Gradle brings the artifacts (packages) into its cache, we let python build tools take over and finish the build. So, any kind of replacement would need to provide a similar functionality.

We improved pyvi-importer behavior in #138 and are working on a better importer that would fully support the corresponding PEP.

Do notice that probably the best option is what we do at LinkedIn. We have a Python application that uploads external packages into our internal artifactory. This app is able to handle even non-standard packages because it's written in Python and can process the code in setup.py that affects the build which Java based programs cannot -- if the package metadata is wrong and relies on Python handling the difference, Java program cannot process it. That said, our Python importer relies on the internal structure and the pypi index in the internal artifactory. We don't have these facilities guaranteed in the open-source artifactory and that's why this Python importer was not open-sourced with pygradle.

We are aware that not every user of pygradle can/wants to use artifactory and that's why we're looking into improving the importer. Such an importer will solve most of the cases correctly (and the improved pivy-importer now does a much better job at it after #138), but there's always a risk of packages using python code instead of metadata in setup.py and dependencies not being recorded correctly because of that.

In short, we'll try to improve this story for the users who cannot rely on the enterprise type artifactory.

[vivekpathak]

Its even worse that some of the pypi artifacts can not be used even in the hackish way described in the iris classifier example. For example:

14:02:21.309 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in opencv-python:3.3.0.10
Exception in thread "main" java.lang.RuntimeException: Unable to find source dist for opencv-python:3.3.0.10
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
	at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:74)
	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
	at com.linkedin.python.importer.deps.DependencyDownloader.downloadDependency(DependencyDownloader.groovy:69)
	at com.linkedin.python.importer.deps.DependencyDownloader$downloadDependency.callCurrent(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:166)
	at com.linkedin.python.importer.deps.DependencyDownloader.download(DependencyDownloader.groovy:55)
	at com.linkedin.python.importer.ImporterCLI.main(ImporterCLI.java:74)

[zvezdan]

@vivekpathak I think this is not our fault. The package you referenced opencv-python does not provide the sdist (source distribution) on PyPI: https://pypi.python.org/pypi/opencv-python/3.3.0.10 Please file the issue with the owners of that package. Package owners should know better and always publish an sdist on PyPI (*.tar.gz file).

On a side note, the workaround that the author of iris example used is not necessary since version 0.5.1. We now use a dependency graph post-order by default (https://github.com/linkedin/pygradle/pull/133). In fact, forcing sorted = false can create more issues than it solves with newer versions of setuptools, wheel, and pip that we started using since version 0.6.1 (https://github.com/linkedin/pygradle/pull/145) if a package that requires pbr for its setup is anywhere in transitive dependencies.

[urisun]

I am trying to use pygradle with my own artifactory server and i created a pypi repo in that server. But there is no ivy metadata. Is there a way to use pivy-importer to generate ivy-metadata and push it to server?

Basically i am looking for the missing link specified by sholsapp in this comment "... we cannot use pypi directly.

... because PyPI currently doesn't host the Ivy metadata that Gradle needs to do proper dependency resolution.

From the same document, there is a link to read more about how to use the import tool that we provide to generate that Ivy metadata. You can configure your own Artifactory-hosted PyPI and use the import tool to generate and publish the Ivy metadata to your own Artifactory."

[Kindrat]

What if whl would be allowed if some platform type is specified via plugin properties? So when there is no sdist artifact whl could be used directly.

[vivekpathak]

Yes - indeed it is not a pygradle problem. In fact if you try to build a pex via pants you get a similar problem.

After some thinking I realized that pants/pex and by consequence pygradle is not equivalent to standard packaging, say by making a wheel. My workaround was to build a wheel - which is more of a first class packaging process being supported as part of python.

I would have liked to use pygradle - but I think we have to fix pants/pex to be able to package whenever we can make a wheel. How to go about that?