Segmentation fault with `hpricot`

[soultoru]

# Enviroment
uname -a      
Linux ip-10-1-25-22 4.4.0-1037-aws #46-Ubuntu SMP Wed Sep 27 19:05:49 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial

ruby -v
ruby 2.3.4p301 (2017-03-30 revision 58214) [x86_64-linux]

# Gemfile
source "https://rubygems.org"
gem 'pycall'
gem 'hpricot'

# open console
LIBPYTHON=/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1 bundle console

irb(main):001:0> require 'pycall/import'
irb(main):002:0> include PyCall::Import
irb(main):003:0> pyimport :math
/mnt/files/apps/test1/vendor/bundle/ruby/2.4.0/gems/pycall-1.0.3/lib/pycall/pyobject_wrapper.rb:71: [BUG] Segmentation fault at 0x0000000000000020
ruby 2.4.3p205 (2017-12-14 revision 61247) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0042 p:---- s:0232 e:000231 CFUNC  :from_ruby
c:0041 p:0033 s:0227 e:000226 METHOD /mnt/files/apps/test1/vendor/bundle/ruby/2.4.0/gems/pycall-1.0.3/lib/pycall/pyobject_wrapper.rb:71
c:0040 p:0026 s:0222 e:000221 METHOD /mnt/files/apps/test1/vendor/bundle/ruby/2.4.0/gems/hpricot-0.8.6/lib/hpricot/blankslate.rb:59 [FINISH]
c:0039 p:-11785133056130 s:0217 e:000216 TOP    [FINISH]
c:0038 p:---- s:0214 e:000213 CFUNC  :require
c:0037 p:0012 s:0209 e:000208 BLOCK  /mnt/files/apps/test1/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292
c:0036 p:0057 s:0206 e:000205 METHOD /mnt/files/apps/test1/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258
c:0035 p:0017 s:0200 e:000199 METHOD /mnt/files/apps/test1/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292
c:0034 p:0021 s:0194 e:000193 RESCUE /mnt/files/apps/test1/vendor/bundle/ruby/2.4.0/gems/pycall-1.0.3/lib/pycall/init.rb:45
c:0033 p:0214 s:0190 e:000189 METHOD /mnt/files/apps/test1/vendor/bundle/ruby/2.4.0/gems/pycall-1.0.3/lib/pycall/init.rb:41
c:0032 p:0071 s:0182 e:000181 METHOD /mnt/files/apps/test1/vendor/bundle/ruby/2.4.0/gems/pycall-1.0.3/lib/pycall/init.rb:16 [FINISH]
c:0031 p:0011 s:0177 e:000176 METHOD /mnt/files/apps/test1/vendor/bundle/ruby/2.4.0/gems/pycall-1.0.3/lib/pycall.rb:52
c:0030 p:0037 s:0172 e:000171 METHOD /mnt/files/apps/test1/vendor/bundle/ruby/2.4.0/gems/pycall-1.0.3/lib/pycall/import.rb:18
c:0029 p:0008 s:0164 e:000163 EVAL   (irb):3 [FINISH]
c:0028 p:---- s:0161 e:000160 CFUNC  :eval

[mrkn]

I've confirmed that Python's math module cannot be imported after hpricot.

With hpricot:

$ irb
irb(main):001:0> require 'pycall'
=> true
irb(main):002:0> math = PyCall.import_module('math')
=> <module 'math' from '/Users/mrkn/.pyenv/versions/3.6.1/Python.framework/Versions/3.6/lib/python3.6/lib-dynload/math.cpython-36m-darwin.so'>
irb(main):003:0>

With hpricot:

$ irb
irb(main):001:0> require 'hpricot'
=> true
irb(main):002:0> require 'pycall'
=> true
irb(main):003:0> math = PyCall.import_module('math')
Traceback (most recent call last):
       11: from /Users/mrkn/.rbenv/versions/2.5/bin/irb:11:in `<main>'
       10: from (irb):3
        9: from /Users/mrkn/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/pycall-1.0.3/lib/pycall.rb:52:in `import_module'
        8: from /Users/mrkn/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/pycall-1.0.3/lib/pycall/init.rb:16:in `const_missing'
        7: from /Users/mrkn/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/pycall-1.0.3/lib/pycall/init.rb:41:in `init'
        6: from /Users/mrkn/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/pycall-1.0.3/lib/pycall/init.rb:45:in `rescue in init'
        5: from /Users/mrkn/.rbenv/versions/2.5.0/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        4: from /Users/mrkn/.rbenv/versions/2.5.0/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        3: from /Users/mrkn/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/hpricot-0.8.6/lib/hpricot/blankslate.rb:59:in `method_added'
        2: from /Users/mrkn/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/pycall-1.0.3/lib/pycall/pyobject_wrapper.rb:71:in `!='
        1: from /Users/mrkn/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/pycall-1.0.3/lib/pycall/pyobject_wrapper.rb:71:in `from_ruby'
FrozenError (can't modify frozen NilClass)
irb(main):004:0>

[phensalves]

Hi @mrkn any solution? Same problem here.

[archonic]

I'm also getting a segmentation fault. I'm not sure if it has to do with a python math library.

/usr/local/bundle/gems/pycall-1.0.0/lib/pycall/pyobject_wrapper.rb:43: [BUG] Segmentation fault at 0x0000000000000008
ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0039 p:---- s:0205 e:000204 CFUNC  :diff
c:0038 p:0102 s:0198 e:000197 METHOD /usr/local/bundle/gems/pycall-1.0.0/lib/pycall/pyobject_wrapper.rb:43
c:0037 p:0013 s:0190 e:000189 METHOD /usr/local/bundle/bundler/gems/ruby_html_tree_differ-c7ad2fc24d6e/lib/ruby_html_tree_differ.rb:10
c:0036 p:0142 s:0184 E:001070 METHOD /app/app/workers/document_changes_relay_worker.rb:24
c:0035 p:0009 s:0172 E:0010f0 METHOD /usr/local/bundle/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:185

ruby_html_tree_differ is just a ruby wrapper using pycall to call a html_tree_diff function. It consistently works after a reboot, then consistently segfaults on the second run, bringing my sidekiq server down, which is kind of a big problem 😅

[mrkn]

Oh, I 'm surprised that hpricot is still being used in production. As three reports came, so I will seriously address this issue. But, unfortunately, perhaps until I finish RubyKaigi in April, I think I can not spare time for this problem.

[archonic]

I'm not using hpricot but I can consistently reproduce the issue. I can make an example app to reproduce if it helps.

[mrkn]

@archonic Could you try with pycall 1.0.3?

[jeremyhaile]

We're getting segmentation faults (not with hpricot) and I was looking at some of the issues. I'm surprised this issue is "Open" but hasn't had a comment since 2019?

[mrkn]

@jeremyhaile Thank you for remembering me this old issue!