Ferrum script has process_timeout errors after upgrading to the Heroku Chrome-for-testing buildpack

[kanejamison]

Describe the bug

Our Ferrum script starts having process_timeout errors after upgrading to the Heroku/Chrome-for-testing buildpack. The old heroku/google-chrome buildpack is deprecated in favor of chrome-for-testing so we were testing this before moving to their new Heroku-24 stack.

I originally posted this as an issue in the chrome-for-testing buildpack repo since it was related to migrating the buildpack, but after confirming that which chrome and which chromedriver were finding the correct paths, it feels relevant to post it here instead.

To Reproduce

I switched over from heroku/google-chrome this week and followed the instructions on Migrating from Separate Buildpacks (though I used the Heroku app settings UI to remove and add the new buildpacks instead of CLI):

Since then we have a cron job running screenshots using Ferrum that has started failing with the following error:

ERROR taking screenshot: Browser did not produce websocket url within 10 seconds, try to increase ":process_timeout". See https://github.com/rubycdp/ferrum#customization

Actions I've taken to try and resolve:

• I have increased our :process_timeout as recommended in the error notes. New setting is 20 seconds and error continued.
• I've read the other Buildpack readme notes and can't see a clear change that would affect Ferrum - they do mention that certain prior buildpack shims for --headless are gone and that app paths have changed, but I confirmed that which chrome and which chromedriver were finding the correct paths, and I'm assuming that Ferrum is already passing any '--headless' types of flags since it runs headless by default.
• I've restarted dynos and deployed a couple times since the buildpack was changed.
• I've tried setting BROWSER_PATH ENV VAR to /app/.chrome-for-testing/chrome-linux64/chrome in case the path wasn't getting picked up somehow.
• I manually moved the chrome-for-testing buildpack into first position in the Heroku UI to confirm it wasn't an issue with buildpack order.
• Can't find anyone else talking about the buildpack change on Twitter or Google searches.
• I've confirmed that buildpack seems to be building OK in build log:

-----> Chrome for Testing app detected
-----> Installing Chrome for Testing
       Resolved STABLE version 128.0.6613.119
       Downloading Chrome
       Downloading Chromedriver
-----> Installing Chrome dependencies

Relevant app settings:

• Rails 6.1
• Ruby 3.1
• Heroku-22 stack
• Ferrum 0.15 (latest)
• 2x standard dynos

Here's the script I'm running that is failing in case it's helpful. We don't have a specific Ferrum config outside of this call to Ferrum::Browser.new(). The final rescue StandardError => e is where my exception is getting tracked.

# frozen_string_literal: true

module ScreenshotCaptureable
  extend ActiveSupport::Concern

  included do
    BROWSER_WINDOW_SIZE = [1200, 900]

    def filename
      @filename ||= ["screenshot", self.class.name, id, Time.now.to_i].compact.map(&:to_s).map(&:parameterize).join("-")
    end

    def file_path
      @file_path ||= "public/screenshot-#{self.class.name}-#{self.id}.jpeg"
    end

    def save_screenshot_to_s3
      screenshot.attach({filename: "#{filename}.png", io: File.open(file_path), content_type: "image/jpeg"})
      self.update_columns(screenshot_updated_at: Time.current)
    end

    def capture_screenshot_via_ferrum(context)
      page = context.create_page
      url = "#{ENV['BASE_URL']}/l/#{special_screenshot_link&.uuid}/template?headless=1"
      Rails.logger.info "#{'*' * 50} #{url.inspect} #{'*' * 50}"

      page.go_to(url)
      page.mouse.scroll_to(10, 60)
      page.screenshot(path: file_path)
    end

    def self.update_next_batch_pending_screenshots
      if next_batch_for_screenshot.count == 0
        Rails.logger.info "No Templates to capture screenshots"
        return
      end

      begin
        next_batch_for_screenshot.each {|model| model.create_sharing_links unless model.readonly_sharing_link&.uuid }

        browser = Ferrum::Browser.new(slowmo: 6, window_size: BROWSER_WINDOW_SIZE, timeout: 15, process_timeout: 20)
        context = browser.contexts.create
        threads = []

        Rails.logger.info "Processing next #{next_batch_for_screenshot.count} Templates"

        next_batch_for_screenshot.each do |model|
          threads << Thread.new(context) do |c|
            begin
              model.capture_screenshot_via_ferrum(context)
              model.save_screenshot_to_s3
            rescue StandardError => e
              Rails.logger.error e.message
            end
          end
        end

        threads.each(&:join)
        context.dispose # dispose the context
        browser.quit # quit the browser instance
      rescue StandardError => e
        Rails.logger.error e.message
        Honeybadger.notify("ERROR taking screenshot: #{e.message}")
      end
    end
  end

end