pygradle
pygradle copied to clipboard
linkedin
Nexus PyPi repositories support
Basically, that's not a feature request - rather a success story. Don't know if dirty hacks, used to add Nexus support are worth adding to upstream, so wanted to describe everything first.
Original task Build and deploy tensorflow-based artifacts with Gradle
Problems:
- [Broken PyPi packages] packages with unresolvable/missing/incorrect dependencies (most scientific related)
- [Missing support for custom (ivy) artifacts in twine] The simplest way to deploy converted with ivy-importer artifacts is twine (
find ${repo} -type f -exec twine upload -r nexus {} \;). But twine supports only certain artifact extensions. - [Nexus breaks python naming convention] https://issues.sonatype.org/browse/NEXUS-12075
- [Gradle does not expose API for custom repositories] https://github.com/gradle/gradle/issues/1400
Solutions:
- Added lenient import for pivy-importer (https://github.com/linkedin/pygradle/pull/185). It helped with a large set of transient dependencies with the only small percent of broken packages. Broken were imported after fixing them manually.
- Added support for
ivyartifact types to twine. https://github.com/pypa/twine/pull/300 - Created Gradle plugin to address naming convention issue - to replace
.with-and to convert to lower case module part of a link path.
Plugin code is ugly, uses reflection and patches internal API of Gradle repository management, but it works.
Structure:
buildSrc
|_src
|_main
|_groovy
|_package
|-NexusPyPiPlugin
|-NexusPyPiRepository
|-NexusPyPiResolver
|_NexusPyPiResourcePattern
Sources:
class NexusPyPiPlugin implements Plugin<Project> {
static final String nexusModulePattern = '[nexus_module]'
@Override
void apply(Project project) {
DefaultRepositoryHandler handler = project.repositories
Field repositoryFactoryField = DefaultRepositoryHandler.getDeclaredField('repositoryFactory')
repositoryFactoryField.setAccessible(true)
DefaultBaseRepositoryFactory repositoryFactory = repositoryFactoryField.get(handler) as DefaultBaseRepositoryFactory
Field instantiatorField = DefaultBaseRepositoryFactory.getDeclaredField('instantiator')
instantiatorField.setAccessible(true)
// Add a method nexusPyPi to the repositories that uses our subclass of the ivy repository.
handler.metaClass.nexusPypi = { Closure closure ->
NexusPyPiRepository artifactRepository = new NexusPyPiRepository(
repositoryFactory.fileResolver,
repositoryFactory.transportFactory,
repositoryFactory.locallyAvailableResourceFinder,
repositoryFactory.artifactFileStore,
repositoryFactory.externalResourcesFileStore,
repositoryFactory.createAuthenticationContainer(),
repositoryFactory.ivyContextManager,
repositoryFactory.moduleIdentifierFactory,
repositoryFactory.instantiatorFactory,
repositoryFactory.fileResourceRepository,
repositoryFactory.metadataParser,
repositoryFactory.experimentalFeatures,
repositoryFactory.ivyMetadataFactory
)
artifactRepository.layout("pattern", new Action<IvyPatternRepositoryLayout>() {
@Override
void execute(IvyPatternRepositoryLayout repositoryLayout) {
repositoryLayout.artifact(nexusModulePattern + '/[revision]/[module]-[revision].[ext]')
repositoryLayout.ivy(nexusModulePattern + '/[revision]/[module]-[revision].ivy')
repositoryLayout.setM2compatible(true)
}
})
handler.addRepository(artifactRepository, 'pypi', ConfigureUtil.configureUsing(closure))
}
}
}
class NexusPyPiRepository extends DefaultIvyArtifactRepository {
private static final Object[] NO_PARAMS = new Object[0]
private final
static Factory<ComponentMetadataSupplier> NO_METADATA_SUPPLIER = new Factory<ComponentMetadataSupplier>() {
@Override
ComponentMetadataSupplier create() {
return null
}
}
private final InstantiatorFactory instantiatorFactory
private final FileStore<String> externalResourcesFileStore
private final RepositoryTransportFactory transportFactory
private final FileResolver fileResolver
private final LocallyAvailableResourceFinder<ModuleComponentArtifactMetadata> locallyAvailableResourceFinder
private final FileStore<ModuleComponentArtifactIdentifier> artifactFileStore
private final AuthenticationContainer authenticationContainer
private final IvyContextManager ivyContextManager
private final ImmutableModuleIdentifierFactory moduleIdentifierFactory
private final ModuleMetadataParser metadataParser
private final IvyArtifactRepository.MetadataSources metadataSources
private final IvyMutableModuleMetadataFactory metadataFactory
private final FileResourceRepository fileResourceRepository
private Class<? extends ComponentMetadataSupplier> componentMetadataSupplierClass
private Object[] componentMetadataSupplierParams
NexusPyPiRepository(
FileResolver fileResolver,
RepositoryTransportFactory transportFactory,
LocallyAvailableResourceFinder<ModuleComponentArtifactMetadata> locallyAvailableResourceFinder,
FileStore<ModuleComponentArtifactIdentifier> artifactFileStore,
FileStore<String> externalResourcesFileStore,
AuthenticationContainer authenticationContainer,
IvyContextManager ivyContextManager,
ImmutableModuleIdentifierFactory moduleIdentifierFactory,
InstantiatorFactory instantiatorFactory,
FileResourceRepository fileResourceRepository,
ModuleMetadataParser metadataParser,
ExperimentalFeatures experimentalFeatures,
IvyMutableModuleMetadataFactory metadataFactory) {
super(
fileResolver,
transportFactory,
locallyAvailableResourceFinder,
artifactFileStore,
externalResourcesFileStore,
authenticationContainer,
ivyContextManager,
moduleIdentifierFactory,
instantiatorFactory,
fileResourceRepository,
metadataParser,
experimentalFeatures,
metadataFactory)
this.metadataFactory = metadataFactory
this.metadataParser = metadataParser
this.fileResourceRepository = fileResourceRepository
this.moduleIdentifierFactory = moduleIdentifierFactory
this.ivyContextManager = ivyContextManager
this.authenticationContainer = authenticationContainer
this.artifactFileStore = artifactFileStore
this.locallyAvailableResourceFinder = locallyAvailableResourceFinder
this.fileResolver = fileResolver
this.transportFactory = transportFactory
this.externalResourcesFileStore = externalResourcesFileStore
this.instantiatorFactory = instantiatorFactory
Field metadataSourcesField = DefaultIvyArtifactRepository.getDeclaredField('metadataSources')
metadataSourcesField.setAccessible(true)
metadataSources = metadataSourcesField.get(this) as IvyArtifactRepository.MetadataSources
}
@Override
protected IvyResolver createRealResolver() {
URI uri = getUrl()
Field layoutField = DefaultIvyArtifactRepository.getDeclaredField('layout')
layoutField.setAccessible(true)
AbstractRepositoryLayout layout = layoutField.get(this) as AbstractRepositoryLayout
Set<String> schemes = new LinkedHashSet<String>()
Field additionalPatternsLayoutField = DefaultIvyArtifactRepository.getDeclaredField('additionalPatternsLayout')
additionalPatternsLayoutField.setAccessible(true)
AbstractRepositoryLayout additionalLayout = additionalPatternsLayoutField.get(this) as AbstractRepositoryLayout
layout.addSchemes(uri, schemes)
additionalLayout.addSchemes(uri, schemes)
IvyResolver resolver = createResolver(schemes)
if (uri != null) {
resolver.addArtifactLocation(uri, IVY_ARTIFACT_PATTERN)
resolver.addDescriptorLocation(uri, IVY_ARTIFACT_PATTERN)
}
layout.apply(uri, resolver)
additionalLayout.apply(uri, resolver)
return resolver
}
void setMetadataSupplier(Class<? extends ComponentMetadataSupplier> ruleClass) {
this.componentMetadataSupplierClass = ruleClass
this.componentMetadataSupplierParams = NO_PARAMS
}
@Override
void setMetadataSupplier(Class<? extends ComponentMetadataSupplier> rule, Action<? super ActionConfiguration> configureAction) {
DefaultActionConfiguration configuration = new DefaultActionConfiguration()
configureAction.execute(configuration)
this.componentMetadataSupplierClass = rule
this.componentMetadataSupplierParams = configuration.getParams()
}
private IvyResolver createResolver(Set<String> schemes) {
if (schemes.isEmpty()) {
throw new InvalidUserDataException("You must specify a base url or at least one artifact pattern for an Ivy repository.")
}
return createResolver(transportFactory.createTransport(schemes, getName(), getConfiguredAuthentication()))
}
private IvyResolver createResolver(RepositoryTransport transport) {
Instantiator instantiator = createDependencyInjectingInstantiator(transport)
return new NexusPyPiResolver(getName(), transport, locallyAvailableResourceFinder,
getResolve().isDynamicMode(), artifactFileStore, moduleIdentifierFactory,
createComponentMetadataSupplierFactory(instantiator), createMetadataSources())
}
private ImmutableMetadataSources createMetadataSources() {
def sources = new ArrayList<>()
if (metadataSources.gradleMetadata) {
sources.add(new DefaultGradleModuleMetadataSource(metadataParser, metadataFactory, true))
}
if (metadataSources.ivyDescriptor) {
sources.add(new DefaultIvyDescriptorMetadataSource(IvyMetadataArtifactProvider.INSTANCE, createIvyDescriptorParser(), fileResourceRepository, moduleIdentifierFactory))
}
if (metadataSources.artifact) {
sources.add(new DefaultArtifactMetadataSource(metadataFactory))
}
return new DefaultImmutableMetadataSources(Collections.unmodifiableList(sources))
}
private MetaDataParser<MutableIvyModuleResolveMetadata> createIvyDescriptorParser() {
return new IvyContextualMetaDataParser<MutableIvyModuleResolveMetadata>(ivyContextManager, new IvyXmlModuleDescriptorParser(new IvyModuleDescriptorConverter(moduleIdentifierFactory), moduleIdentifierFactory, fileResourceRepository, metadataFactory))
}
/**
* Creates a service registry giving access to the services we want to expose to rules and returns an instantiator that
* uses this service registry.
* @param transport the transport used to create the repository accessor
* @return a dependency injecting instantiator, aware of services we want to expose
*/
private Instantiator createDependencyInjectingInstantiator(final RepositoryTransport transport) {
DefaultServiceRegistry registry = new DefaultServiceRegistry()
registry.addProvider(new Object() {
RepositoryResourceAccessor createResourceAccessor() {
return createRepositoryAccessor(transport)
}
})
return instantiatorFactory.inject(registry)
}
private RepositoryResourceAccessor createRepositoryAccessor(RepositoryTransport transport) {
return new ExternalRepositoryResourceAccessor(getUrl(), transport.getResourceAccessor(), externalResourcesFileStore)
}
private Factory<ComponentMetadataSupplier> createComponentMetadataSupplierFactory(final Instantiator instantiator) {
if (componentMetadataSupplierClass == null) {
return NO_METADATA_SUPPLIER
}
return new Factory<ComponentMetadataSupplier>() {
@Override
ComponentMetadataSupplier create() {
return instantiator.newInstance(componentMetadataSupplierClass, componentMetadataSupplierParams)
}
}
}
}
class NexusPyPiResolver extends IvyResolver {
NexusPyPiResolver(
String name,
RepositoryTransport transport,
LocallyAvailableResourceFinder<ModuleComponentArtifactMetadata> locallyAvailableResourceFinder,
boolean dynamicResolve,
FileStore<ModuleComponentArtifactIdentifier> artifactFileStore,
ImmutableModuleIdentifierFactory moduleIdentifierFactory,
Factory<ComponentMetadataSupplier> componentMetadataSupplierFactory,
ImmutableMetadataSources metadataSources) {
super(
name,
transport,
locallyAvailableResourceFinder,
dynamicResolve,
artifactFileStore,
moduleIdentifierFactory,
componentMetadataSupplierFactory,
metadataSources,
IvyMetadataArtifactProvider.INSTANCE)
}
@Override
void addArtifactLocation(URI baseUri, String pattern) {
addArtifactPattern(toResourcePattern(baseUri, pattern))
}
@Override
void addDescriptorLocation(URI baseUri, String pattern) {
addIvyPattern(toResourcePattern(baseUri, pattern))
}
private static ResourcePattern toResourcePattern(URI baseUri, String pattern) {
return new NexusPyPiResourcePattern(baseUri, pattern)
}
}
class NexusPyPiResourcePattern extends M2ResourcePattern {
NexusPyPiResourcePattern(URI baseUri, String pattern) {
super(baseUri, pattern)
}
protected String substituteTokens(String pattern, Map<String, String> attributes) {
String module = attributes.get("module");
if (module != null) {
attributes.put("nexus_module", module.replace(".", "-").toLowerCase());
}
return super.substituteTokens(pattern, attributes);
}
}
Usage example:
apply plugin: "com.linkedin.python"
apply plugin: NexusPyPiPlugin
python {
forceVersion('pypi', 'pip', '9.0.1')
testDir = file('tests')
}
repositories {
nexusPypi {
url = "https://mynexus/repository/pypi-releases/packages/"
if (project.hasProperty('repository.username')) {
credentials.username = project.property('repository.username')
}
if (project.hasProperty('repository.password')) {
credentials.password = project.property('repository.password')
}
}
}
installPythonRequirements {
sorted = false
}
dependencies {
python 'pypi:tensorflow:1.5.0'
}
Created pull request for twine ivy extension https://github.com/pypa/twine/pull/300. Even if it would be rejected, you can still get my patch there
@kopy07 Is this worth another pull request? I mean new plugin for nexus support
Hey @Kindrat,
Cool investigation! You would also probably do
((ProjectInternal)project).getServices().get(BaseRepositoryFactory.class) to get access to the BaseRepositoryFactory class.
If you want to open a PR I'll review it and we can get it in, let's try to making it java as much as possible (trying to kill the main groovy code). Defining the repo can't be done in java as far as I know, but the rest of it should be doable there.
HI ! @Kindrat, Can you explain, what type of repository have you added in Nexus? As I understand, the Nexus supports only Maven repository (not Ivy).
@shvilime Nexus supports all kinds of repositories. The goal was to use existing Nexus instead of deploying additional artifactory instance.
@Kindrat, I didn't find in nexus documentation supported type information about Ivy format. What "existing Nexus" repository do you mean ? What type of repository in Nexus allow to upload *.ivy ?
@shvilime Existing means the one I already use for Npm, Maven and Pypi. As for ivy - gradle allows to define repo url pattern, ivy is one of supported for any type of repo. I'm using Pypi repo with ivy metadata so that is could work as vanilla Pypi and as gradle repo with all features of dependency management.
@Kindrat, thanks for your comment. PyPI repo supports in Nexus since version 3.0.2. OK, I have to upgrade our Nexus. Anyway, if we want to build private repository, we have to prepare ivy and upload the artifactories to Nexus (big job), or proxy Nexus to existing PyPI repository with ivy. Have you tried to proxy Nexus on public linkedin Artifactory repository or something like this?
Nope. I've patched pivy-importer to generate all the ivy metadata for my dep tree and twine to upload whole local pypi-ivy repo to Nexus.
@Kindrat, Can you briefly explain, how to create PyPI repository, that allow to attach ivy files to main artifactory in Nexus 3?
@Kindrat Are you use in Nexus "PyPi type" of repository + additional upload ivy file in each artifact(I can't do this), or "RAW type" of repository that allow upload what you want ?
@shvilime please, read "solutions" section of my original post. I've patched twine to upload ivy files within regular sdists and taught gradle plugin to use pypi-ivy metadata similar to how it uses it for java.