Mixed content issues

[yogodoshi]

Hi, I'm getting a lot of timeout errors when running @browser.network.wait_for_idle and debugging it, seems that the reason is that the https website is trying to make http requests but Ferrum is interpreting those requests as still pending pending? => true instead of saying that they errored or something like that.

Here is some code that might better explain:

@browser.network.traffic[39].pending?
=> true

@browser.network.traffic[39]
=> #<Ferrum::Network::Exchange @id="368.86" @intercepted_request=nil @request=#<Ferrum::Network::Request:0x00007f0ea42bf368 @params={"requestId"=>"368.86", "loaderId"=>"A4955E508B726946F0BE99E45102B73F", "documentURL"=>"https://janeteststore1-rev.myshopify.com/password", "request"=>{"url"=>"http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js", "method"=>"GET", "headers"=>{"Referer"=>""}, "mixedContentType"=>"blockable", "initialPriority"=>"Low", "referrerPolicy"=>"no-referrer-when-downgrade"}, "timestamp"=>936576.281018, "wallTime"=>1600352504.098881, "initiator"=>{"type"=>"script", "stack"=>{"callFrames"=>[{"functionName"=>"", "scriptId"=>"27", "url"=>"https://www.papernap.com/assets/shopify-rev/js/shopify-airs-embed.js?shop=janeteststore1-rev.myshopify.com", "lineNumber"=>8, "columnNumber"=>41}]}}, "type"=>"Script", "frameId"=>"93980CA9F68B59F23BDA3F8BBD5B50F8", "hasUserGesture"=>false}, @request={"url"=>"http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js", "method"=>"GET", "headers"=>{"Referer"=>""}, "mixedContentType"=>"blockable", "initialPriority"=>"Low", "referrerPolicy"=>"no-referrer-when-downgrade"}> @response=nil @error=nil>

All the other requests that return false on pending? access https urls, the two that are http return true.

If instead of saying that it is still pending, it returned an error (like what we see in "mixedContentType"=>"blockable"), the timeout exception wouldn't be raised.

Another thing to not is that acessing that page using my OS X Chrome browser, the request works fine, it doesn't raise any error on the console, so another possibility might be a config to ignore mixed content issues?

Thanks for the work put in this gem :)

[route]

Hey, do you have a web page I can test this behavior on?

[yogodoshi]

@route not mine but here it is: https://janeteststore1-rev.myshopify.com/password

Not suggesting replicating this code, as you need to decide if it makes sense for the gem and what's the best approach to solve it but here is a monkey patch I quickly implemented to workaround it:

module Ferrum
  class Network
    class Exchange
      def finished?
        blocked? || response || error || blocked_because_of_mixed_content
      end

      def blocked_because_of_mixed_content
        request.inspect.include?('"mixedContentType"=>"blockable"')
      end
    end
  end
end

[balt5r]

@yogodoshi might be related to #124.

Did you find solution? For now I removed the error from wait_for_idle

module Ferrum
  class Network
    def wait_for_idle(connections: 0, duration: 0.05, timeout: @page.browser.timeout)
      start = Ferrum.monotonic_time

      until idle?(connections)
        break if Ferrum.timeout?(start, timeout)
        sleep(duration)
      end
    end
  end
end

[yogodoshi]

@balt5r no solution but I changed a different part:

module Ferrum
  class Network
    class Exchange
      # https://github.com/rubycdp/ferrum/blob/7950d5dbb889ed8e40eca4cbc974677b9acaba1c/lib/ferrum/network/exchange.rb
      def finished?
        blocked? || response || error || blocked_because_of_mixed_content
      end

      def blocked_because_of_mixed_content
        request.inspect.include?('"mixedContentType"=>"blockable"')
      end
    end
  end
end

[route]

It looks like for your request @error=nil and thus it shows up as pending. I just tested this behaviour in latest Chrome and looks like it's working because now the error is set correctly, for example:

#<Ferrum::Network::Error:0x000000010505c688 @canceled=false, @error_text="", @id="76845.3", @monotonic_time=410616.446262, @type="Stylesheet">

even though the content is mixed:

=> #<Ferrum::Network::Exchange @id="76845.3" @intercepted_request=nil @request=#<Ferrum::Network::Request:0x000000010505d218 @params={"requestId"=>"76845.3", "loaderId"=>"D2FF0540CCA637AA536AC026C141B5C1", "documentURL"=>"https://593e-84-54-122-156.eu.ngrok.io/", "request"=>{"url"=>"http://cdn.jsdelivr.net/npm/[email protected]/dist/css/bootstrap.min.css", "method"=>"GET", "headers"=>{"Referer"=>"", "Origin"=>"https://593e-84-54-122-156.eu.ngrok.io"}, "mixedContentType"=>"blockable", "initialPriority"=>"VeryHigh", "referrerPolicy"=>"strict-origin-when-cross-origin", "isSameSite"=>false}, "timestamp"=>410616.446195, "wallTime"=>1669009169.856324, "initiator"=>{"type"=>"parser", "url"=>"https://593e-84-54-122-156.eu.ngrok.io/", "lineNumber"=>0, "columnNumber"=>215}, "redirectHasExtraInfo"=>false, "type"=>"Stylesheet", "frameId"=>"AE059DDE99B1A116E7D53CD4AAFB6DFE", "hasUserGesture"=>false}, @request={"url"=>"http://cdn.jsdelivr.net/npm/[email protected]/dist/css/bootstrap.min.css", "method"=>"GET", "headers"=>{"Referer"=>"", "Origin"=>"https://593e-84-54-122-156.eu.ngrok.io"}, "mixedContentType"=>"blockable", "initialPriority"=>"VeryHigh", "referrerPolicy"=>"strict-origin-when-cross-origin", "isSameSite"=>false}> @response=nil @error=#<Ferrum::Network::Error:0x000000010505c688 @id="76845.3", @type="Stylesheet", @error_text="", @monotonic_time=410616.446262, @canceled=false>>

Let me know if it doesn't work in current master branch and latest Chrome.