prepare-offline-pack returns HTTP 407 when using proxy with basic auth

[jguay]

• Version: LS 6.5.6 and 6.1.2
• Operating System: CentOS 7
• Steps to Reproduce: docker_squidAndLogstashWithIptables.zip

1- Create an image from logstash image adding iptables (using my docker id):

cd logstash612-with-iptables
docker image build --tag logstash612-with-iptables .
cd ..

2- run

docker-compose up

$ docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                              NAMES
7b7e55df75d3        logstash612-with-iptables:latest   "/usr/local/bin/dock…"   35 minutes ago      Up 13 minutes       5044/tcp, 0.0.0.0:9600->9600/tcp   logstash612
75dd6fc2ff22        robhaswell/squid-authenticated     "/init"                  21 hours ago        Up 13 minutes       3128/tcp                           squid

3- login to terminal as root :

docker exec -i -t --privileged -u root logstash612 /bin/bash

4- Block access to outisde ports 80 and 443 (REJECT or DROP):

iptables -A OUTPUT -p tcp -m tcp -m multiport --dports 80,443 -j REJECT

5- Verify connection only works via proxy :

echo $https_proxy
curl https://www.elastic.co
env -i curl https://www.elastic.co

The last command show that without using env variable https_proxy defined in docker-compose.yml, direct connection to 443 fails... 6- Install plugin (via proxy) :

JARS_SKIP='true' JRUBY_OPTS='-J-Dhttps.proxyHost=elastic:change@squid -J-Dhttps.proxyPort=3128 -J-Dhttp.proxyHost=elastic:changeme@squid -J-Dhttp.proxyPort=3128' DEBUG=1 /usr/share/logstash/bin/logstash-plugin install logstash-filter-aggregate

This works OK 4- Preparing offline package fails :

JARS_SKIP='true' JRUBY_OPTS='-J-Dhttps.proxyHost=elastic:change@squid -J-Dhttps.proxyPort=3128 -J-Dhttp.proxyHost=elastic:changeme@squid -J-Dhttp.proxyPort=3128' DEBUG=1 /usr/share/logstash/bin/logstash-plugin prepare-offline-pack logstash-filter-aggregate

This fails to use proxy credentials :

DEBUG: exec /usr/share/logstash/vendor/jruby/bin/jruby /usr/share/logstash/lib/pluginmanager/main.rb prepare-offline-pack logstash-filter-aggregate
[INFO]: Cleaning existing target path: /tmp/studtmp-d94a3dde1fa6e434fdad542faf61bca0e24f9e08ae7d1fe0541c9275bf66
[INFO]: Vendoring: logstash-filter-aggregate-2.7.2.gem, downloading: https://rubygems.org/downloads/logstash-filter-aggregate-2.7.2.gem
Net::HTTPServerException: 407 "Proxy Authentication Required"

Note the access.log of squid shows the same error for an attempt to use proxy without password

Note proxy_support.rb contain these values if I iterate through the proxy_settings hash:

-------------https proxy_settings---------------
protocol https
host squid
port 3128
username elastic
password changeme
---------------------------

[nicolargo]

Same issue here with Logstash 6.2.3.

$ logstash-plugin prepare-offline-pack --overwrite --output logstash-plugins.zip logstash-output-influxdb
Net::HTTPServerException: 407 "Proxy Authentication Required"
                 error! at /home/nhe/logstash-6.2.3/vendor/jruby/lib/ruby/stdlib/net/http/response.rb:120
                  value at /home/nhe/logstash-6.2.3/vendor/jruby/lib/ruby/stdlib/net/http/response.rb:129
                connect at /home/nhe/logstash-6.2.3/vendor/jruby/lib/ruby/stdlib/net/http.rb:925
               do_start at /home/nhe/logstash-6.2.3/vendor/jruby/lib/ruby/stdlib/net/http.rb:868
                  start at /home/nhe/logstash-6.2.3/vendor/jruby/lib/ruby/stdlib/net/http.rb:857
                request at /home/nhe/logstash-6.2.3/vendor/jruby/lib/ruby/stdlib/net/http.rb:1409
                    get at /home/nhe/logstash-6.2.3/vendor/jruby/lib/ruby/stdlib/net/http.rb:1167
          download_file at /home/nhe/logstash/vendor/bundle/jruby/2.3.0/gems/paquet-0.2.1/lib/paquet/utils.rb:21
           download_gem at /home/nhe/logstash/vendor/bundle/jruby/2.3.0/gems/paquet-0.2.1/lib/paquet/gem.rb:106
  block in package_gems at /home/nhe/logstash/vendor/bundle/jruby/2.3.0/gems/paquet-0.2.1/lib/paquet/gem.rb:48
                   each at org/jruby/RubyArray.java:1734
           package_gems at /home/nhe/logstash/vendor/bundle/jruby/2.3.0/gems/paquet-0.2.1/lib/paquet/gem.rb:42
                   pack at /home/nhe/logstash/vendor/bundle/jruby/2.3.0/gems/paquet-0.2.1/lib/paquet/gem.rb:33
                execute at /home/nhe/logstash/lib/pluginmanager/offline_plugin_packager.rb:88
                package at /home/nhe/logstash/lib/pluginmanager/offline_plugin_packager.rb:115
                execute at /home/nhe/logstash/lib/pluginmanager/prepare_offline_pack.rb:41
                    run at /home/nhe/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67
                execute at /home/nhe/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/subcommand/execution.rb:11
                    run at /home/nhe/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67
                    run at /home/nhe/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132
                 <main> at /home/nhe/logstash/lib/pluginmanager/main.rb:48

Note: Installation works like a charm:

$ logstash-plugin install logstash-input-stomp
Validating logstash-input-stomp
Installing logstash-input-stomp
Installation successful

[michael-doubez]

I got it working with prepare-offline-pack by modifying vendor/bundle/jruby/2.3.0/gems/paquet-0.2.1/lib/paquet/utils.rb.

I basically added support for https_proxy but I guess there may be a more elegant way by using configure_proxy result and injecting the proxy informations

def self.download_file(source, destination, counter = REDIRECTION_LIMIT)
...
    uri = URI.parse(source)

    # Get proxy information
    proxy_url = ENV["https_proxy"] || ENV["HTTPS_PROXY"] || ""
    proxy_uri = URI(proxy_url)

    http = Net::HTTP.new(uri.host, uri.port, proxy_uri.host, proxy_uri.port)
    http.proxy_user = proxy_uri.user
    http.proxy_pass = proxy_uri.password

    http.use_ssl = uri.scheme ==  HTTPS_SCHEME

    response = http.get(uri.path)
    ....

[nicolargo]

Any head up ?

[makefu]

Just a heads-up the issue persists until today ( Logstash-8.1.0 OSS flavor ). the patch provided by @michael-doubez works

[kaisecheng]

issue from net-http https://github.com/ruby/net-http/issues/68