cannot load such file -- robots (LoadError) with ruby verison 2.5

[bartman081523]

wayback_archiver Traceback (most recent call last): 10: from /usr/local/bin/wayback_archiver:23:in <main>' 9: from /usr/local/bin/wayback_archiver:23:in load' 8: from /var/lib/gems/2.7.0/gems/wayback_archiver-1.4.0/bin/wayback_archiver:4:in <top (required)>' 7: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 6: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 5: from /var/lib/gems/2.7.0/gems/wayback_archiver-1.4.0/lib/wayback_archiver.rb:4:in <top (required)>' 4: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 3: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 2: from /var/lib/gems/2.7.0/gems/wayback_archiver-1.4.0/lib/wayback_archiver/url_collector.rb:2:in <top (required)>' 1: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require': cannot load such file -- robots (LoadError) 11: from /usr/local/bin/wayback_archiver:23:in

' 10: from /usr/local/bin/wayback_archiver:23:in load' 9: from /var/lib/gems/2.7.0/gems/wayback_archiver-1.4.0/bin/wayback_archiver:4:in <top (required)>' 8: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 7: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 6: from /var/lib/gems/2.7.0/gems/wayback_archiver-1.4.0/lib/wayback_archiver.rb:4:in <top (required)>' 5: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 4: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 3: from /var/lib/gems/2.7.0/gems/wayback_archiver-1.4.0/lib/wayback_archiver/url_collector.rb:2:in <top (required)>' 2: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:149:in require' 1: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:160:in rescue in require' /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:160:in require': cannot load such file -- robots (LoadError) user@tp-ubuntu:~/Downloads/wayback_archiver$ sudo gem install robots Successfully installed robots-0.10.1 Parsing documentation for robots-0.10.1 Done installing documentation for robots after 0 seconds 1 gem installed user@tp-ubuntu:~/Downloads/wayback_archiver$ wayback_archiver Traceback (most recent call last): 10: from /usr/local/bin/wayback_archiver:23:in

' 9: from /usr/local/bin/wayback_archiver:23:in load' 8: from /var/lib/gems/2.7.0/gems/wayback_archiver-1.4.0/bin/wayback_archiver:4:in <top (required)>' 7: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 6: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 5: from /var/lib/gems/2.7.0/gems/wayback_archiver-1.4.0/lib/wayback_archiver.rb:4:in <top (required)>' 4: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 3: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 2: from /var/lib/gems/2.7.0/gems/wayback_archiver-1.4.0/lib/wayback_archiver/url_collector.rb:2:in <top (required)>' 1: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require': cannot load such file -- robots (LoadError) 11: from /usr/local/bin/wayback_archiver:23:in <main>' 10: from /usr/local/bin/wayback_archiver:23:in load' 9: from /var/lib/gems/2.7.0/gems/wayback_archiver-1.4.0/bin/wayback_archiver:4:in <top (required)>' 8: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 7: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 6: from /var/lib/gems/2.7.0/gems/wayback_archiver-1.4.0/lib/wayback_archiver.rb:4:in <top (required)>' 5: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 4: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' 3: from /var/lib/gems/2.7.0/gems/wayback_archiver-1.4.0/lib/wayback_archiver/url_collector.rb:2:in <top (required)>' 2: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:149:in require' 1: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:160:in rescue in require' /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:160:in require': cannot load such file -- robots (LoadError) user@tp-ubuntu:~/Downloads/wayback_archiver$ ruby --version ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux-gnu]

RESOLUTION: (funtioning, but not valid anymore, see my last comment)

sudo apt purge ruby
sudo apt autoremove
sudo apt install curl
sudo apt install git
command curl -sSL https://rvm.io/mpapis.asc | gpg --import -
command curl -sSL https://rvm.io/pkuczynski.asc | gpg --import -
\curl -sSL https://get.rvm.io | bash -s stable
source /home/$USER/.rvm/scripts/rvm
rvm install ruby-3
git clone https://github.com/buren/wayback_archiver
cd wayback_archiver
gem build wayback_archiver.gemspec
gem install wayback_archiver-1.4.0.gem 
wayback_archiver --url www.example.com
echo "source /home/$USER/.rvm/scripts/rvm" >> /home/$USER/.bashrc

[buren]

@chlorophyll-zz this doesn't seem like this is an issue with wayback_archiver?

[bartman081523]

@buren I dont know. /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:160:in require': cannot load such file -- robots (LoadError)

I think this means that the robots module is not available in ruby 2.7?

I just wanted to have the resolution for users affected here. The resolution is to install ruby 3, which is unfortunately not avialable for debian ?stable?. So I have this resolution to install ruby 3 with rvm.

Could you restrict the version of ruby to 3? Or test the gem with ruby 2.7?

[buren]

I think this means that the robots module is not available in ruby 2.7?

robots is a gem (referenced here in wayback_archiver.gemspec)

It works for me on Ruby 2.7 👇

@chlorophyll-zz so there must be something else that is causing issues? 🤔

[bartman081523]

@buren Thank you. I am going to close this. I only have investigated this because #46 was mentioning that ruby version. I probably was running into that error when I tested with ruby 2.7. Btw can you close #46, this is completely resolved, but the user did not close the issue. EDIT: The error was with ruby 2.5, which I tested because the user from #46 did not upgrade to 2.7. So I also was wrong there. Ruby 2.7 is available for debian oldstable. Never mind.

[buren]

Thanks!

[bartman081523]

I have this Issue again with ruby 3.0 in Ubuntu Jammy. I would prefer to not install ruby through rvm and keep the packaged ruby.

$ wayback_archiver 
<internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:160:in `require': cannot load such file -- robots (LoadError)
	from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
	from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:149:in `require'
	from /var/lib/gems/3.0.0/gems/wayback_archiver-1.4.0/lib/wayback_archiver/url_collector.rb:2:in `<top (required)>'
	from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from /var/lib/gems/3.0.0/gems/wayback_archiver-1.4.0/lib/wayback_archiver.rb:4:in `<top (required)>'
	from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from /var/lib/gems/3.0.0/gems/wayback_archiver-1.4.0/bin/wayback_archiver:4:in `<top (required)>'
	from /usr/local/bin/wayback_archiver:25:in `load'
	from /usr/local/bin/wayback_archiver:25:in `<main>'
<internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- robots (LoadError)
	from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from /var/lib/gems/3.0.0/gems/wayback_archiver-1.4.0/lib/wayback_archiver/url_collector.rb:2:in `<top (required)>'
	from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from /var/lib/gems/3.0.0/gems/wayback_archiver-1.4.0/lib/wayback_archiver.rb:4:in `<top (required)>'
	from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from /var/lib/gems/3.0.0/gems/wayback_archiver-1.4.0/bin/wayback_archiver:4:in `<top (required)>'
	from /usr/local/bin/wayback_archiver:25:in `load'
	from /usr/local/bin/wayback_archiver:25:in `<main>'

gem list

...
robots (0.10.1)
...

Edit: The error is only with robots v 0.10.1, it is working with robots 0.10.0

[bartman081523]

Issue is still pending, #64 is the fix

[dewey]

I think the best fix would be to vendor the one robots.rb file from the library as it seems the gem build process was just buggy, the code itself is fine. That's how metasploit did it: https://github.com/rapid7/metasploit-framework/commit/0e1bafb2d123019a42fe2524d897c0df2aeba4f8