JBake fails to run or build on Apple Silicon

[mhberger]

Hi there

Thank you for JBake – I have been using it since June 2015 (that is when I get the urge to write something pithy at Code is Mandatory)

Recently I started using a MacBook Air (with Apple M1 Silicon) for processing and unfortunately JBake does not run or build on Apple M1 Silicon. It works fine on Apple Intel.

Exceptions and references to the underlying issue are listed below.

Digging in it is appears to be an issue with OrientDB.

As a result of nosing around and to solve this itch, I have started work on removing the dependency of OrientDB and using JDBC and Groovy Sql with the implementation done in SQLite.

The work is ongoing on a “Prototype PR” type branch that is really rough as guts at the moment.

Exceptions caused trying to run

$ jbake -b
JBake v2.7.0-rc.2 (2021-05-24 22:30:41[GMT+01:00] 045c61c#) [http://jbake.org]

14:02:31.341 INFO  org.jbake.app.Oven - Baking has started...
Warning: Nashorn engine is planned to be removed from a future JDK release
14:02:31.485 INFO  c.o.common.jna.ONative - Default limit of open files (512) will be used.
14:02:31.518 INFO  c.o.common.jna.ONative - 17179869184 B/16384 MB/16 GB of physical memory were detected on machine
14:02:31.518 INFO  c.o.common.jna.ONative - Detected memory limit for current process is 17179869184 B/16384 MB/16 GB
14:02:31.519 INFO  c.o.o.c.e.OMemoryAndLocalPaginatedEnginesInitializer - JVM can use maximum 4096MB of heap memory
14:02:31.519 INFO  c.o.o.c.e.OMemoryAndLocalPaginatedEnginesInitializer - Because OrientDB is running outside a container 12% of memory will be left unallocated according to the setting 'memory.leftToOS' not taking into account heap memory
14:02:31.519 INFO  com.orientechnologies - OrientDB auto-config DISKCACHE=10,321MB (heap=4,096MB os=16,384MB)
14:02:31.519 INFO  c.o.o.c.e.l.OEngineLocalPaginated - System is started under an effective user : `mark`
14:02:31.520 INFO  c.o.o.c.e.l.OEngineLocalPaginated - Allocation of 156879 pages.
14:02:31.529 ERROR jbake - An unexpected error occurred: /private/var/folders/43/bt5j2vf520v6n9gnx968cs840000gn/T/jna-3344077/jna5212213165608407388.tmp: dlopen(/private/var/folders/43/bt5j2vf520v6n9gnx968cs840000gn/T/jna-3344077/jna5212213165608407388.tmp, 1): no suitable image found.  Did find:
	/private/var/folders/43/bt5j2vf520v6n9gnx968cs840000gn/T/jna-3344077/jna5212213165608407388.tmp: no matching architecture in universal wrapper
	/private/var/folders/43/bt5j2vf520v6n9gnx968cs840000gn/T/jna-3344077/jna5212213165608407388.tmp: no matching architecture in universal wrapper
mhbmacxiv jbake (master) $

Exceptions caused trying to run tests

GroovyTemplateEngineRenderingTest > renderTags FAILED
    org.gradle.internal.exceptions.DefaultMultiCauseException: Multiple Failures (3 failures)
    	java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
    	java.lang.NullPointerException: Can't set default locale to NULL

Related Git Hub Issues

Test failures on Apple silicon (aarch64) #1323 Add darwin-aarch64 support #1297 Enable building jna for Darwin arm64 #1238 Support for Apple silicon #20 Fix variadic arguments on arm64 darwin ABI #577 Enable Strict Type Enforcement for Dynamic Method Dispatching

Software environment

# uname -a
Darwin mhbmacxiv 20.5.0 Darwin Kernel Version 20.5.0: Sat May  8 05:10:31 PDT 2021; root:xnu-7195.121.3~9/RELEASE_ARM64_T8101 arm64

# sw_vers
ProductName:	macOS
ProductVersion:	11.4
BuildVersion:	20F71

# java --full-version
openjdk 11.0.11+9-LTS


# jbake version (running lastest snapshot)
JBake v2.7.0-rc.2 (2021-05-24 22:30:41[GMT+01:00] 045c61c#) [http://jbake.org]

# jbake version (trying to build)
$ git log | head
commit 109b7c1379b907172acfee7815d945d3c1c5c51b
Merge: 045c61c 5829541
Author: Jonathan Bullock <[email protected]>
Date:   Tue Jun 8 22:48:33 2021 +0100

    Merge pull request #707 from jonbullock/fix/692-template-folders

    Fix to allow templates to be stored in folders

[jbaron]

Facing same issue, is there already some idea/agreement if moving to sqlite for future releases might be the right approach?

@mhberger saw that you continue to update your fork. Is that fork more or less usable on M1 ?

[mhberger]

Hi. In reply to about whether it is more or less usable, it builds (using SQLite) on M1, Intel, Linux and Windows.

I use it mainly on M1. Once I unzip the installaiton and set path, it works building my blog and the JBake website.

# Build
./gradlew clean smoketest assemble

# The distributions are in jbake-dist/build/distributions. Unzip jbake-2.7.0-rc.4-bin.zip and add
# jbake-2.7.0-rc.4-bin/bin to path and then you should be able to use jbake
#

But to get it polished and merged back into master/main will require some tidying up and and deciding on what to do for the future.

Currently the work on the Prototype/PR branch includes the following

• Uses JDBC and SQLite insted of OrientDB
• Upgraded to use Gradle 7
• Upgraded to use JDK 11
• Upgraded to use JUnit 5
• Got it working with TeamCity Cloud Beta. But then once the beta finiished, had to stop – the monthly ongoing cost is a bit steep.
• Got it (sort of) working with Azure Pipelines, but have not pursued this.

[jbaron]

Thanks, just tried it out and compiles and runs fine on my M1 and even feels faster (btw my website uses asciidoc + freemarker)

But it seems it doesn't handle custom doc types yet. But this could also be a 2.7.x compatibility issue since previously I used the maven plugin and I guess that is still on the 2.6.x version. Need to invest a bit more.

[mhberger]

K. Have you got an example using custom doc types? It could be that I missed something moving away from OrientDB.

[jbaron]

I guess the easiest way would be to clone my website (small one):

git clone https://github.com/neurallayer/roboquant.org.git
cd roboquant.org
jbake . docs

But if a dedicated example is more useful, just let me know and I'll make one.

[mhberger]

That is great. Thanks. Let me take a look and I will see if I can figure it out.

[jbaron]

P.S the default type is a custom type. So these two lines in the jbake.properties files are relevant:

template.doc.file=doc.ftl
default.type=doc

[mhberger]

Thanks again. I understand what is supposed to happen. And I can see in the code where it is “failing”.

I will find the functionality and associated test (if there is not one will try and put something together).

[jbaron]

That's is quick, thanks!!! If there are some test or samples that required that I can help with, just let me know.

[mhberger]

Doh! I missed a call to updateDocTypesFromConfiguration(). This has been fixed and a test added. I have just used the templates and a page from your website as the test resource. Let me know if this does not suit. Or if we can simplify.

[jbaron]

Just tested it also and it works great!!!

My website generates correctly and also very fast. Possible part of this is due to the M1 chip, but I also think SQLite might have less overhead, especially for smaller sites.

[jbaron]

@jonbullock Would you consider moving (or supporting) JBake to SQLLite in the future, or are there obstacles in doing so?

[mhberger]

Hi.

I have created a new branch (mhb/sqlite-2022-01-04) off the old one (mhb/sqlite-2021-05-30) and then applied new changes from master.

The use case I have been focusing on is unzipping the dist zip and running that. i.e. I have not tested/used the plugins, mainly because I don’t use maven day to day. Any pointers to how to test/use the plugins more than welcome.

Have tested it on Apple M1 and Apple Intel with my blog and with jbake.org and seems to be working okay.

I need to think of a way of tidying up the branch to make it easier to understand the changes. Something like this article

Updated: fixed typo.

[jonbullock]

Apologies for the delay in replying to this.

First off let me say thanks to you both for spending time looking into this. I'm not adverse to supporting a different content store other than OrientDB (maybe via a config option so end users have a choice?). However, a non-relational data store (Document support in OrientDB) was selected to enable users to have flexibility in the schema when it comes to content and it's metadata. I've not looked at the SQLite implementation yet but this is the major consideration that came to mind. If SQLite didn't support a flexible schema this would be a hurdle as it would negatively impact users.

[jbaron]

I also didn't look into the details of the SQL Lite implementation, but I'm having some custom content types and properties and they seem to be working fine. A simple key/value pair table in traditional RDBMS can go a long way to support flexible schemes ;)

P.S For me one of the biggest benefit besides it is now also running on Apple M1, is the speed improvements.

[jonbullock]

Very true, could be solved in that way.

From the brief research I've done the root cause of the issue seems to be in JNA that OrientDB uses, but I can't find any reported issue in the OrientDB project. I'll raise one to see what the project maintainers say about it. I hadn't realised a fork of OrientDB had occurred recently too.

How much improvement are you seeing with SQLite?

[jbaron]

I didn’t time the improvements, but if I have some time will give it a go. But from just watching the console output it was clearly noticeable.

[mhberger]

Hi Jon. Thanks for feedback.

re SQLite/JDBC. Although it is a relational, it is being used as a content store similar to the OrientDB approach, with custom types being stored in a JSON blob. Part of the work was to create a ContentStore interface and use that.

The ContentStore interface should allow changing implementations. Have also used JDBC and Groovy SQL and DataSource to provide a bit more independence. It should be relatively painless to switch to another implementation.

At runtime, which contentstore to use is controlled via jbake.properties by setting jbake_db_implementation=SQLite or jbake_db_implementation=OrientDB. The default on the branch is SQLite.

The DocumentModel is still the main interface between parsers/templates etc and the ContentStore. It is translated to a Document which maps to the underlying DB table structure.

re speed. This requires care when comparing. One has to take account the speed improvement due to Apple Silicon. The following gives some idea building a simple blog and jbake.org.

Site	Hardware	Database	Bake Info
simple blog	iMac Pro Intel	OrientDB	Baked 47 items in 4044ms
simple blog	iMac Pro Intel	SQLite	Baked 47 items in 2839ms
simple blog	MacBook Pro M1 Max	SQLite	Baked 47 items in 1836ms
jbake.org	iMac Pro Intel	OrientDB	Baked 196 items in 10731ms
jbake.org	iMac Pro Intel	SQLIte	Baked 196 items in 15461ms
jbake.org	MacBook Pro M1 Max	SQLite	Baked 196 items in 9611ms

I am going to work on tidying up the commits probably organising them into something along the following lines:

• introduce db.package
• introduce a ContenStore interface
• add sqlite as a database engine
• remove deprecated methods
• convert junit tests to use junit5
• add integration tests

[lvca]

Hi guys, OrientDB is not supported anymore. We are working on ArcadeDB, a fork of OrientDB with an active community and a new engine that is much lighter and faster than OrientDB maintaining the same SQL and similar API. Also, ArcadeDB supports Cypher and GraphQL in case you need it, and much more.

Last but not least, it's embeddable in Java, exactly like OrientDB and it has zero or minimal dependencies based on the configuration used.

If you're interested I'd be happy to help in the update to ArcadeDB, just give me some hint where the repository layer is located and I can do a quick POC ;-)

[jonbullock]

Hi @mhberger

Thanks for the stats regarding speed.

It's more the custom metadata per content file than custom types of content I'm concerned about: https://jbake.org/docs/2.6.7/#custom_metadata

[jonbullock]

Hi @lvca

The big question is does ArcadeDB work on Apple Silicon?

[lvca]

Hi @jonbullock yes it does. It's 100% pure Java, there is no JNI, JNA, or other esoteric configuration like OrientDB ;-)

[lvca]

I've replaced OrientDB with ArcadeDB in the PR https://github.com/jbake-org/jbake/pull/745. It was easy and quick, the SQL is the same, just a little change in the API.

[lprimak]

Same issue here, a blocker because these M1 macs are exploding in popularity

[lprimak]

I did manage a workaround by patching JNA in jbake with the latest JNA distribution

[kwin]

@jonbullock Is this supposed to be fixed with the upcoming 2.7.0?

[lprimak]

Simplest way to resolve this is to just upgrade the versions of JNA in the next JBake release

[kwin]

@lprimak Thanks for the hint, can you raise a PR?

[lprimak]

@kwin probably not in a timely fashion

[kwin]

OrientDB migrated from JNA to JNR in https://github.com/orientechnologies/orientdb/commit/74276df8dbb14fc413c08e2fed91dcf527fad797 and was recently updated to a version supporting Apple Silicon (https://github.com/orientechnologies/orientdb/pull/9762). The most recent release 3.1.16 should contain that fix (https://github.com/orientechnologies/orientdb/commits/3.1.x)