Support authentication to private maven repositories (Artifactory)

[wstrange]

I tried to use a private Artifactory repository that requires authentication.

I have the appropriate credentials in my ~/.m2/settings.xml, but it looks like the resolver does not reference these credentials. The http request for the artifact gets a 401 unauthorized response.

[akashdayal]

Hope its not too late but you can use https://:@ as resolver for private repositories.

[wstrange]

Does that imply the credentials would be in in the clear in the generated workspace.bzl? Is there any trick to pick up the credentials from an external source (.m2/settings.xml would be ideal.)

Thanks!

[johnynek]

Maybe we could look in an environmental variable for Auth or ask the user to enter auth at s prompt?

We use certificates at Stripe so I have not been in this situation.

On Thu, Apr 19, 2018 at 06:41 Warren Strange [email protected] wrote:

Does that imply the credentials would be in in the clear in the generated workspace.bzl? Is there any trick to pick up the credentials from an external source (.m2/settings.xml would be ideal.)

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/johnynek/bazel-deps/issues/95#issuecomment-382802685, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEJduJBdjuuzXZOLwh0aKBRcyWMpzeOks5tqL49gaJpZM4QbHRk .

-- P. Oscar Boykin, Ph.D. | http://twitter.com/posco | http://pobox.com/~boykin

[diacone]

Hope its not too late but you can use https://:@ as resolver for private repositories.

This doesn't work either. Even though I can clearly see that url contains username/password:

java.io.IOException: Error downloading [https://<user>:<passwd>@my.private.repo/content/repositories/privaterepo/com/google/guava/guava/19.0/guava-19.0-sources.jar] to /home/myuser/.cache/bazel/_bazel_myuser/4d81720667242c59d08111d8c78d97a8/external/com_google_guava_guava/jar/com_google_guava_guava-sources.jar: GET returned 401 Unauthorized

[deadmoose]

I've seen just that with our setup; the repo doesn't cope with auth in the url, it really has to go into headers.

I've been using a hack described in https://github.com/bazelbuild/bazel/issues/5452#issuecomment-432739019

TLDR: Flag private repos and fall back to shelling out to mvn & letting it handle authentication for me.

It's not a great approach & not in a form I'd consider floating it as a patch, but it's been good enough to unblock us for now.

[diacone]

I patched the workspace to shell out to curl, but it still requires plaintext password in deps.yml and workspace itself, but at least i'm unblocked.

[guw]

@diacone Are you open to share your patch? I'm in a similar spot trying to fetch from a password protected Nexus.

[diacone]

@diacone Are you open to share your patch? I'm in a similar spot trying to fetch from a password protected Nexus.

diff --git a/src/scala/com/github/johnynek/bazel_deps/templates/jar_artifact_backend.bzl b/src/scala/com/github/johnynek/bazel_deps/templates/jar_artifact_backend.bzl
index c4bc52f..b8f58fe 100644
--- a/src/scala/com/github/johnynek/bazel_deps/templates/jar_artifact_backend.bzl
+++ b/src/scala/com/github/johnynek/bazel_deps/templates/jar_artifact_backend.bzl
@@ -1,22 +1,14 @@
 def _jar_artifact_impl(ctx):
     jar_name = "%s.jar" % ctx.name
-    ctx.download(
-        output=ctx.path("jar/%s" % jar_name),
-        url=ctx.attr.urls,
-        sha256=ctx.attr.sha256,
-        executable=False
-    )
-    src_name="%s-sources.jar" % ctx.name
-    srcjar_attr=""
+    result = ctx.execute(["curl", ctx.attr.urls[0], "-o", ctx.path("jar/%s" % jar_name), "--create-dirs"])
+    if result.return_code:
+        fail(result.stderr)
+    src_name = "%s-sources.jar" % ctx.name
+    srcjar_attr = ""
     has_sources = len(ctx.attr.src_urls) != 0
     if has_sources:
-        ctx.download(
-            output=ctx.path("jar/%s" % src_name),
-            url=ctx.attr.src_urls,
-            sha256=ctx.attr.src_sha256,
-            executable=False
-        )
-        srcjar_attr ='\n    srcjar = ":%s",' % src_name
+        ctx.execute(["curl", ctx.attr.urls[0], "-o", ctx.path("jar/%s" % src_name), "--create-dirs"])
+        srcjar_attr = '\n    srcjar = ":%s",' % src_name
 
     build_file_contents = """
 package(default_visibility = ['//visibility:public'])

[guw]

Thanks a lot @diacone. Just to confirm, this diff is for the "runtime" build part, i.e. when Bazel is downloading dependencies? It does not deal with the "resolution" part, i.e. when bazel-deps is supposed to resolve artifacts from a privat Nexus server which requires authentication?

[diacone]

Thanks a lot @diacone. Just to confirm, this diff is for the "runtime" build part, i.e. when Bazel is downloading dependencies? It does not deal with the "resolution" part, i.e. when bazel-deps is supposed to resolve artifacts from a privat Nexus server which requires authentication?

Yes @guw, this is for the runtime part.

[johnynek]

For resolution, I’d rather prompt for a password if a flag is given, since I’d rather not be checking in auth credentials.

I can help you get started on a patch if you like. The first thing to look at would be making an optional flag to ask the user for user/password. Then we can see about passing authentication data to the resolver.

[guw]

@johnynek Thanks! I'm certainly interested in providing a patch. I'm converting a fairly large legacy project. So far I'm running into resolution issue with Coursier and trying Aether now.

Prompting is not something that's going to work as we need to run this on CI. Ideally, I'd like to re-use authentication information stored in ~/.m2/settings.xml. Bazel's native maven_jar seems to support this ootb. Thus, I was wondering if I can supply a different template which would use native.maven_jar for download.

[johnynek]

Interesting that you run resolution on CI. We don’t do that since it is pretty slow.

For downloading on build, you can pretty easily change how resolution works by adjusting the callback .bzl function that processes the resolved artifacts.

[johnynek]

See: https://github.com/johnynek/bazel-deps/blob/6585033409e09028852403ec15ec0c77851234be/3rdparty/workspace.bzl#L188

You can pass any function you want in which can use any method you need to set up the dependencies after resolution.

[guw]

FYI: Bazel respository_ctx.download currently doesn't support auth. https://github.com/bazelbuild/bazel/issues/7469

[AlterEgo7]

Any progress now that bazelbuild/bazel#7469 is resolved?

[guw]

FWIW, the issue became less pressing for us since we started using https://github.com/salesforce/bazel-maven-proxy/

[johnynek]

Happy to accept a PR, but I am personally not working on this issue.

I'm also happy to answer questions and guide someone through making the PR