Narray build issue

[ajkotobi]

Hi @masa16,

I'm trying to build narray latest version and last previous 3 releases with

Ruby-2.6.1
GCC-9.1.0

but all fails. I've pasted build logs from Spack package manager

==> Installing ruby-narray
==> No binary for ruby-narray found: installing from source
==> Fetching https://github.com/ruby-numo/numo-narray/archive/v0.9.1.8.tar.gz
######################################################################## 100.0%
==> ruby-narray: Executing phase: 'build'
==> [2020-11-09-11:04:22.024157] '/sw/spack-rhel6/ruby-2.6.2-p33cxs/bin/gem' 'build' '--norc' 'numo-narray.gemspec'
fatal: Not a git repository (or any of the parent directories): .git
WARNING:  open-ended dependency on rake (>= 12.3.3, development) is not recommended
  if rake is semantically versioned, use:
    add_development_dependency 'rake', '~> 12.3', '>= 12.3.3'
WARNING:  See http://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: numo-narray
  Version: 0.9.1.8
  File: numo-narray-0.9.1.8.gem
==> ruby-narray: Executing phase: 'install'
==> [2020-11-09-11:04:22.418063] '/sw/spack-rhel6/ruby-2.6.2-p33cxs/bin/gem' 'install' '--norc' '--ignore-dependencies' 'numo-narray-0.9.1.8.gem'
Building native extensions. This could take a while...
ERROR:  Error installing numo-narray-0.9.1.8.gem:
	ERROR: Failed to build gem native extension.

    current directory: /mnt/lustre01/spack-workplace/spack-rhel6/ruby-narray-0.9.1.8-ufp3le/gems/numo-narray-0.9.1.8/ext/numo/narray
/mnt/lustre01/spack-workplace/spack-rhel6/ruby-2.6.2-p33cxs/bin/ruby -I /mnt/lustre01/spack-workplace/spack-rhel6/ruby-2.6.2-p33cxs/lib/ruby/2.6.0 -r ./siteconf20201109-37266-1kzysu3.rb extconf.rb
checking for stdbool.h... yes
checking for stdint.h... yes
checking for bool in stdbool.h... yes
checking for u_int8_t in stdint.h... yes
checking for u_int16_t in stdint.h... yes
checking for int32_t in stdint.h... yes
checking for u_int32_t in stdint.h... yes
checking for int64_t in stdint.h... yes
checking for u_int64_t in stdint.h... yes
checking for exp10()... yes
checking for rb_arithmetic_sequence_extract()... yes
checking for rb_cComplex... yes
creating /mnt/lustre01/spack-workplace/spack-rhel6/ruby-narray-0.9.1.8-ufp3le/gems/numo-narray-0.9.1.8/ext/numo/narray/numo/extconf.h
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/mnt/lustre01/spack-workplace/spack-rhel6/ruby-2.6.2-p33cxs/bin/$(RUBY_BASE_NAME)
/mnt/lustre01/spack-workplace/spack-rhel6/ruby-2.6.2-p33cxs/lib/ruby/2.6.0/mkmf.rb:1716:in `initialize': No such file or directory @ rb_sysopen - /mnt/lustre01/spack-workplace/spack-rhel6/ruby-narray-0.9.1.8-ufp3le/gems/numo-narray-0.9.1.8/ext/numo/narray/numo/extconf.h (Errno::ENOENT)
	from /mnt/lustre01/spack-workplace/spack-rhel6/ruby-2.6.2-p33cxs/lib/ruby/2.6.0/mkmf.rb:1716:in `open'
	from /mnt/lustre01/spack-workplace/spack-rhel6/ruby-2.6.2-p33cxs/lib/ruby/2.6.0/mkmf.rb:1716:in `create_header'
	from extconf.rb:96:in `<main>'

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /mnt/lustre01/spack-workplace/spack-rhel6/ruby-narray-0.9.1.8-ufp3le/extensions/x86_64-linux/2.6.0-static/numo-narray-0.9.1.8/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /mnt/lustre01/spack-workplace/spack-rhel6/ruby-narray-0.9.1.8-ufp3le/gems/numo-narray-0.9.1.8 for inspection.
Results logged to /mnt/lustre01/spack-workplace/spack-rhel6/ruby-narray-0.9.1.8-ufp3le/extensions/x86_64-linux/2.6.0-static/numo-narray-0.9.1.8/gem_make.out
==> Error: ProcessError: Command exited with status 1:
    '/sw/spack-rhel6/ruby-2.6.2-p33cxs/bin/gem' 'install' '--norc' '--ignore-dependencies' 'numo-narray-0.9.1.8.gem'

1 error found in build log:
     44    	--without-opt-lib=${opt-dir}/lib
     45    	--with-make-prog
     46    	--without-make-prog
     47    	--srcdir=.
     48    	--curdir
     49    	--ruby=/mnt/lustre01/spack-workplace/spack-rhel6/ruby-2.6.2-p33cxs/
           bin/$(RUBY_BASE_NAME)
  >> 50    /mnt/lustre01/spack-workplace/spack-rhel6/ruby-2.6.2-p33cxs/lib/ruby
           /2.6.0/mkmf.rb:1716:in `initialize': No such file or directory @@ rb_
           sysopen - /mnt/lustre01/spack-workplace/spack-rhel6/ruby-narray-0.9.
           1.8-ufp3le/gems/numo-narray-0.9.1.8/ext/numo/narray/numo/extconf.h (
           Errno::ENOENT)
     51    	from /mnt/lustre01/spack-workplace/spack-rhel6/ruby-2.6.2-p33cxs/li
           b/ruby/2.6.0/mkmf.rb:1716:in `open'
     52    	from /mnt/lustre01/spack-workplace/spack-rhel6/ruby-2.6.2-p33cxs/li
           b/ruby/2.6.0/mkmf.rb:1716:in `create_header'
     53    	from extconf.rb:96:in `<main>'
     54    
     55    To see why this extension failed to compile, please check the mkmf.l
           og which can be found here:
     56    

I assume it fails during objs creation

$objs = srcs.collect{|i| i+".o"} <---

create_header d('numo/extconf.h')

Any ideas how to tackle this?

Thanks

[kojix2]

Hi You tried to install this package, right? https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/ruby-narray/package.py

[ajkotobi]

Hi,

Yes That package.

Thanks

[kojix2]

Thank you. @ajkotobi

Workaround:. You might be able to install numo-narray using the gem command installed by spack. For example

sudo /home/kojix2/spack/opt/spack/linux-ubuntu20.04-zen/gcc-9.3.0/ruby-2.7.1-atsdaekpplyrurpxpc3h4jlvhxvabgbw/bin/gem install numo-narray

The installation of the Ruby package for spack is described here. https://github.com/spack/spack/blob/develop/lib/spack/spack/build_systems/ruby.py

However, you can't install numo-narray in the following way.

wget https://github.com/ruby-numo/numo-narray/archive/v0.9.1.8.tar.gz
tar -xvf v0.9.1.8.tar.gz
cd numo-narray-0.9.1.8/
gem build numo-narray.gemspec
gem install numo-narray-0.9.1.8.gem

To generate extconf.h, you need the following command

ruby ext/numo/narray/extconf.rb

or use rake command.

rake install

~~Parhaps spack is not currently able to install the ruby libraries that use the C extension. Maybe it needs to be fixed a bit.~~

ping @masa16

[kojix2]

Link: https://github.com/spack/spack/issues/19814

[ajkotobi]

Thanks @kojix2.

I've already tried using .../bin/gem install numo-narray but it fails as it requires Spack environment variable 'SPACK_ENV_PATH' which is missing, and I did not go down the path with that, as looking for more generic way.

Using other methods out of Spack environment going to fail at the beginning as

/sw/spack-rhel6/ruby-2.6.2-p33cxs/bin/ruby extconf.rb 
checking for stdbool.h... *** extconf.rb failed ***

I'm able to install other type of ruby based packages, but no luck with narray. I already Opened up an issue with Spack as you linked it in here.

[kojix2]

Sorry, it seems to be a narray issue, not a spack issue.

https://github.com/ruby-numo/numo-narray/pull/164

The problem may be fixed if you could use the latest github master.

[ajkotobi]

Thanks @kojix2

I could successfully compile narray from master branch, and created in Spack PR 19817, so at least package can be compiled from master branch.

[ajkotobi]

Hi @kojix2,

I'm reopening this issue for a question regarding compilation. I've compiled numo-narray from master branch as you pointed that out, but how would that be possible to have it compiled for older versions? as it is needed by our users. Is there any work around it?

As in previous comments mentioned, here is the error e.g. for 0.9.0.9 version

1 error found in build log:
     39    	--without-opt-lib=${opt-dir}/lib
     40    	--with-make-prog
     41    	--without-make-prog
     42    	--srcdir=.
     43    	--curdir
     44    	--ruby=/mnt/lustre01/spack-workplace/spack-rhel6/ruby-2.6.2-p33cxs/
           bin/$(RUBY_BASE_NAME)
  >> 45    /mnt/lustre01/spack-workplace/spack-rhel6/ruby-2.6.2-p33cxs/lib/ruby
           /2.6.0/mkmf.rb:1716:in `initialize': No such file or directory @@ rb_
           sysopen - numo/extconf.h (Errno::ENOENT)

Thanks

[kojix2]

Hi @ajkotobi I'm not a NArray developer and I can't read code written in C, but I can still provide some useful information.

In conclusion, v0.9.0.9 cannot be installed on ruby 2.5 or later.

I checked by switching Ruby versions with rbenv and found that v0.9.0.9 can be installed with Ruby 2.4 but not with Ruby 2.5.

gem install numo-narray -v '0.9.0.9' # install specific version
# Ruby 2.4 : ○
# Ruby 2.5 : ×

But we can insatll v0.9.1.0 on Ruby 2.5

gem install numo-narray -v '0.9.1.0' # install specific version
# Ruby 2.5 : ○

If you look at the commit logs prior to the v0.9.1.0 release, most of them relate to Ruby 2.5 support. Therefore, v0.9.0.9 is not expected to work with Ruby 2.5 or later. See the commit log below..

https://github.com/ruby-numo/numo-narray/compare/v0.9.0.9...v0.9.1.0

The logs you presented show that you are using Ruby 2.6.2. So, it would be difficult to use v0.9.0.9 in the first place.

I'm a bit curious about what workflows/applications are using numo-narray v0.9.0.9 . Most parts of the NArray spec haven't been changed other than bug fixes in the last few years. You can probably upgrade your version of narray and it will work just fine.

[kojix2]

@masa16 How about releasing a new version once before Ruby 3.0 comes out?

[kojix2]

@ajkotobi According to the error message you showed me, the direct reason why you could not install v0.9.0.9 is not Ruby 2.6. I looked into this.

The installation process of Ruby packages in Spack is well documented.

• https://github.com/spack/spack/blob/develop/lib/spack/docs/build_systems/rubypackage.rst

And the Python code to install the ruby package is as follows.

• https://github.com/spack/spack/blob/develop/lib/spack/spack/build_systems/ruby.py

Download source code v0.9.1.8

wget https://github.com/ruby-numo/numo-narray/archive/v0.9.1.8.tar.gz
tar -xvf numo-narray-0.9.1.8.tar.gz
cd numo-narray-0.9.1.8

Then install it just like the way spack do.

gem build numo-narray.gemspec # generate numo-narray-0.9.1.8.gem
gem install numo-narray-0.9.1.8.gem

And get the same error!

numo/extconf.h (Errno::ENOENT)

So, it certainly can't be installed in the usual way...

[ajkotobi]

@kojix2 Thanks for input. I'm facing below error as you mentioned for every version of numo/narray with different Ruby version except from master branch. but some users from our center want older version for usage with netcdf(Not fully aware of their workflow).

numo/extconf.h (Errno::ENOENT)

I did the manual steps but did get the upper error too.

[kojix2]

@ajkotobi Hello. Today, I heard some interesting information from @uwabami. If netcdf is ruby-netcdf, then it depends on the old NArray. The old NArray is completely incompatible with numo-narray. Here is the repository.

https://github.com/masa16/narray

Sorry to bother you, but could you confirm whether what your team really needs is the old version of NArray or the new version of numo-narray? If you need the old NArray, I think you can build it using the above repository. Good luck.

git clone https://github.com/masa16/narray
cd narray
gem build narray.gemspec
gem install narray-0.6.1.2.gem

irb

NArray.int(10).indgen
# => 
# NArray.int(10): 
# [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ]