curb

curb copied to clipboard

Seen again in the log with 0.7.8. Here's some code I use to call multi.perform (the retrying is a little aggressive, that has to do with the particular service I am communicating with):

class Curl::Easy
  attr_accessor :retry_timeout
  attr_accessor :hard_error_retry
  attr_accessor :retries

  attr_accessor :multi

  def retry
    self.retries += 1
    if not @multi.nil?
      @multi.add(self)
    else
      perform()
    end
  end

end


def make_request(url)
  ncurl = Curl::Easy.new(url) do |curl|
    curl.useragent = request.user_agent
    # If a zero-length string is set, then an Accept-Encoding header
    # containing all supported encodings is sent.
    curl.encoding = ""
    #curl.verbose = true
    curl.max_redirects = MAX_REDIRECTS
    curl.connect_timeout = 10
    curl.timeout = 300
    # our options
    curl.retry_timeout = 100.0
    curl.retries = 0
    curl.hard_error_retry = 0

    curl.on_failure do |c,e|
      t = Time.now.to_f.to_s.ljust(14)
      if [404, 400, 500].include?(c.response_code)
        puts "#{t} #{c.object_id}: HTTP response code #{c.response_code}. Retrying in #{curl.retry_timeout}ms (#{curl.retries} retries)..."
        if c.response_code != 503
          if curl.hard_error_retry >= 100
            raise HTTPError, "HTTP response code #{c.response_code} hard error retry: #{curl.hard_error_retry}"
          end
          curl.hard_error_retry += 1
        end
        # sleep a little..
        sleep curl.retry_timeout/1000.0
        curl.retry_timeout += 10
        c.retry
      elsif [Curl::Err::TimeoutError, Curl::Err::GotNothingError].include?(e[0])
        puts "#{t} #{c.object_id}: #{e[1]} (#{e[0]}). Retrying (#{curl.retries} retries)..."
        c.retry
      else
        if not [0, 200].include?(c.response_code)
          if [410, 503].include?(c.response_code)
            success_func.call(c.body_str)
          else
            raise HTTPError, "HTTP response code #{c.response_code}"
          end
        else
          raise *e
        end
      end
    end

    curl.on_success do |c|
      success_func.call(c.body_str)
    end

  end
end

@@curl_multi = Curl::Multi.new
urls.each do |url|
  curl = make_request(url)
  curl.multi = @@curl_multi
  @@curl_multi.add(curl)
end

@@curl_multi.perform    

I modified you script slightly to get it to run https://gist.github.com/751155 i'm not seeing the warning you were getting... it is a fairly harmless warning - it just means when we get to the point of GC, the internal Hash that manages the easy structures was unable to locate the easy handle - this is probably because it was all ready released.

Yeah it doesn't happen every time or anything. In fact it happens very very rarely, and I'm not sure which codepath triggers it. It is fairly harmless, I just thought I'd report it.

yeah, it's definitely valid - i've also spotted it a few times... i added that report because I figured it could be possible... so i 'll keep this ticket open...