Add Digest framework support

[konsolebox]

Hi,

Please consider adding Digest framework with C extension support (ruby/digest.h). There are a couple of gems that work on top of it, including my two gems.

[chrisseaton]

Is digest.h part of the Ruby C extension API? I know there's not much of a formal API, but digest.h seems even more internal than normal.

How are you using the header from a C extension gem, from outside the gem?

[jjyr]

Hi, I can't install digest-sha3 gem on truffleruby(1.0.0-rc2), seems it use ruby/digest.h which truffleruby not supported yet. I don't know too much about C-extension, Maybe you can check this gem (https://github.com/phusion/digest-sha3-ruby)

gem install digest-sha3 Full error log:

gem install digest-sha3
Building native extensions.  This could take a while...
ERROR:  Error installing digest-sha3:
	ERROR: Failed to build gem native extension.

    current directory: /Users/jiangjinyang/.rbenv/versions/truffleruby-1.0.0-rc2/lib/ruby/gems/2.4.0/gems/digest-sha3-1.1.0/ext/digest
/Users/jiangjinyang/.rbenv/versions/truffleruby-1.0.0-rc2/bin/truffleruby -r ./siteconf20180713-62285-35w684.rb extconf.rb
checking for ruby/digest.h... no
checking for rb_str_set_len()... yes
creating Makefile

current directory: /Users/jiangjinyang/.rbenv/versions/truffleruby-1.0.0-rc2/lib/ruby/gems/2.4.0/gems/digest-sha3-1.1.0/ext/digest
make "DESTDIR=" clean

current directory: /Users/jiangjinyang/.rbenv/versions/truffleruby-1.0.0-rc2/lib/ruby/gems/2.4.0/gems/digest-sha3-1.1.0/ext/digest
make "DESTDIR="
compiling KeccakF-1600-reference.c
compiling KeccakNISTInterface.c
compiling KeccakSponge.c
compiling displayIntermediateValues.c
displayIntermediateValues.c:113:40: warning: format string is not a string literal (potentially insecure) [-Wformat-security]
        fprintf(intermediateValueFile, text);
                                       ^~~~
displayIntermediateValues.c:113:40: note: treat the string as an argument to avoid this
        fprintf(intermediateValueFile, text);
                                       ^
                                       "%s", 
1 warning generated.
compiling sha3.c
sha3.c:5:10: fatal error: 'digest.h' file not found
#include "digest.h"
         ^~~~~~~~~~
1 error generated.
make: *** [sha3.bc] Error 1

make failed, exit code 2

[chrisseaton]

I'll have to look into how this header is found in MRI, as it's not part of the normal include directory as far as I can tell.

[eregon]

FWIW it seems there is a fallback in the code if there is no ruby/digest.h, but that doesn't work: https://github.com/phusion/digest-sha3-ruby/commit/f7b055100c1e97b47be71193e528729e59f4017d#r29688007

MRI seems to ship include/ruby-X.Y.Z/ruby/digest.h since at least Ruby 1.9.1, so there is probably some special handling and file copying for this file (ext/digest/digest.h in the MRI repository).

[konsolebox]

@chrisseaton This is currently what I recall and re-observed: Digest is basically composed of three parts, one forms the abstract structure (Digest::Class), another is a module (Digest::Instance) that composes the default implementation of the methods defined in Digest::Class, and Digest::Base which puts them all together. Digest::Base basically is the API. Classes that extend it doesn't have to reimplement every method in Digest::Class. They simply have to create three callback methods: one is the initializer function, which the class can use to initialize its data, another is the updater, which updates a current sum, and another is the finalizer method, which finalizes the result. There is a struct called rb_digest_metadata_t in ruby/digest.h, where the data size, and the callback methods are defined. It is a wrapped data which is shared to Digest::Base through an instance variable of the inheriting class, which is @metadata.

One of my gems makes use of Digest::Base and the struct provided by ruby/digest.h, but another directly inherits Digest::Class and doesn't make use of the API, since it had to add compatible enhancements which when implemented only makes the API provided by Digest::Base redundant, and possibly conflicting. It also made the code simpler. This is why it would be nice if the framework is also not simplified to just Digest::Base.

My gems also check the API version (RUBY_DIGEST_API_VERSION).

[eregon]

Probably the easiest to support usages of ruby/digest.h is to reuse the digest C extension from CRuby. @bjfish gave that a try but it's slower than the current implementation, so we need to figure out why.