Performance regression vs non-Hermes version?

[liamjones]

Description

While investigating https://github.com/realm/realm-js/issues/4328 (which is now resolved), I noticed that performance seemed to have halved on the Hermes builds when importing several thousand objects (see https://github.com/realm/realm-js/issues/4328#issuecomment-1059743396).

Stacktrace & log output

10.20.0-beta.2:
Loaded 6066 Products in 13.883 seconds
Loaded 6066 Products in 14.27 seconds
Loaded 6066 Products in 13.969 seconds

10.20.0-beta.1:
Loaded 6066 Products in 13.923 seconds
Loaded 6066 Products in 13.829 seconds
Loaded 6066 Products in 13.909 seconds

10.13.0:
Loaded 6066 Products in 7.503 seconds
Loaded 6066 Products in 7.485 seconds
Loaded 6066 Products in 7.504 seconds

Can you reproduce a bug?

Yes, always

Reproduction Steps

I don't have time to create a repro right now (sorry!) - I'm just logging the ticket now so I don't forget. I will try and come back with a minimal repro in a week or two.

Version

10.20.0-beta.1 & beta.2

What SDK flavour are you using?

Local Database only

Are you using encryption?

No, not using encryption

Platform OS and version(s)

RN 0.66.3 using JSC (not Hermes!) - iPhone 13 Simulator w/ iOS 15.2, Android AVD Emulator running Android 9.0

[tomduncalf]

Thanks for the report @liamjones – we are aware there are regressions here and will be looking into them as part of the ongoing work on Hermes. We'll update you when we have some more info

[liamjones]

Thanks Tom!

[fronck]

@liamjones I'm working on profiling of the Hermes code. Can you share anything about the characteristics of the import you do? E.g.,

• How large are the objects you import typically, schema-wise?
• How do you bundle the writes (one object per write transaction, multiple objects per write transaction)?
• What types do you use (specifically, do you have a lot of, for example, embedded objects, lists, or complex types like Mixed/Set)?

[liamjones]

Hi @fronck, sorry for the delayed reply, been a busy week!

I've cut down a minimal repro and sent it in to [email protected] so you can see the exact code we're running - look for email titled "Private repro for realm-js issues #4443 & #4328"

[tomduncalf]

Hi @liamjones, I've been able to do some investigation into this this week. It turns out that in the current version of the Hermes engine, it is always performing some timing of all JS/C++ calls, which adds quite considerable overhead in our case: https://github.com/facebook/hermes/blob/v0.11.0/API/hermes/hermes.cpp#L185-L194

With this timing disabled (I modified the Hermes source and compiled my own version of it), I see Realm performance on iOS increase by 2-2.5x in our performance benchmarks, which makes it comparable or faster than JSC for our basic data types.

I've raised this with Meta and they have said that “this is a known problem, and we’re planning to make these timers dynamically configurable (and off by default) in the next release”, so the good news is that hopefully this will be fixed in the next React Native release (and might improve performance of anything else which interacts with Hermes!).

The bad news is that there's not much you can do about it for now, unless you are willing to use a custom version of Hermes in your app (I can tell you how to do so if you are interested).

Some of our data type benchmarks are still slower on Hermes vs. JSC, so I'm looking into this to see if there's anything we can do. For reference, these are the results I'm seeing on iOS with my modified Hermes (a number greater than 1 means Hermes is faster, a number less than 1 means Hermes is slower):

Data type	Hermes performance vs JSC
bool	1.07
int	1.00
float	1.13
double	1.15
string	1.00
decimal128	0.15
objectId	0.47
uuid	0.35
date	0.79
data	1.38
Car	0.78
bool[]	0.36
bool<>	0.38
bool{}	0.40

[liamjones]

Thanks for the info @tomduncalf ! Does this apply to my comparisons though? I was using the Hermes build of Realm but still with the JSC, not Hermes as the JS engine. Even then I was seeing this performance reduction.

[tomduncalf]

Oh interesting @liamjones, I totally missed that, sorry! I guess it would not apply in that case... I'll take a look and see if I can reproduce.

[tomduncalf]

By way of an update @liamjones, I've found that the way we are accessing the native Realm C++ objects in v11 via JSI performs quite a lot slower than the way we do it on master via JSC.

The JSI abstraction layer seems to add some cost, but it's mainly the difference between accessing an object property (which seems to be relatively slow) vs. accessing its "private data" (which is quick).

There is no way with JSI to interact directly with the "private data" of an object that I can see, but we are investigating whether there are other ways we can store and access these objects which might perform better. We'll keep you updated on our findings!

[tradebulls]

@tomduncalf @takameyer We are using a custom react native library already. We can modified the Hermes.cpp file and try it out in IOS. However, I am curious to know how could we achieve the same in Android versions. Can you please help?

[HSReact]

Hi @liamjones, I've been able to do some investigation into this this week. It turns out that in the current version of the Hermes engine, it is always performing some timing of all JS/C++ calls, which adds quite considerable overhead in our case: https://github.com/facebook/hermes/blob/main/API/hermes/hermes.cpp#L185-L194

With this timing disabled (I modified the Hermes source and compiled my own version of it), I see Realm performance on iOS increase by 2-2.5x in our performance benchmarks, which makes it comparable or faster than JSC for our basic data types.

I've raised this with Meta and they have said that “this is a known problem, and we’re planning to make these timers dynamically configurable (and off by default) in the next release”, so the good news is that hopefully this will be fixed in the next React Native release (and might improve performance of anything else which interacts with Hermes!).

The bad news is that there's not much you can do about it for now, unless you are willing to use a custom version of Hermes in your app (I can tell you how to do so if you are interested).

Some of our data type benchmarks are still slower on Hermes vs. JSC, so I'm looking into this to see if there's anything we can do. For reference, these are the results I'm seeing on iOS with my modified Hermes (a number greater than 1 means Hermes is faster, a number less than 1 means Hermes is slower):

Data type Hermes performance vs JSC bool 1.07 int 1.00 float 1.13 double 1.15 string 1.00 decimal128 0.15 objectId 0.47 uuid 0.35 date 0.79 data 1.38 Car 0.78 bool[] 0.36 bool<> 0.38 bool{} 0.40

@tomduncalf Hi Tom, thanks for this, do you happen to have performance benchmarks on read/write times, sync times (thats a big one)...many thanks

[tomduncalf]

@HSReact I don't have any benchmarks on those. The only place we would expect to see any change in performance characteristics is in the JS/C++ interop layer (previously JSC, now JSI/Hermes), so we think that benchmarking property access provides a pretty good proxy measurement for this.

Read/write/sync are all implemented purely in our C++ core, so we wouldn't expect to see any performance regression of the actual internal implementations of those features – but if the JS/C++ interop performance is reduced, you would see reduced read/write/sync throughput regardless as each read/write/sync needs to go between C++ and JS.

Once we solve the regressions as measured with the current test suite, I'd expect that to also solve any regressions seen in read/write/sync. However, it's a great suggestion that we should have tests to benchmark these operations more generally, and we will definitely be looking to expand our peformance testing suite in future.

[mfbx9da4]

Definitely agree that having benchmarks is a great place to start.

Just to add to hermes/JSI branch performance issues, an empty write transaction is taking ~20ms as noted in #3709.

Also I'm finding that to iterate through a collection of about 500 items and access the keys on each item takes ~50ms 🤯 . (The objects in the collection are fairly small trivial objects). For context the same code takes ~1ms in plain JS, 50x faster.

Workarounds would be greatly appreciated, thanks.

[tomduncalf]

I'll document the workaround for when Hermes is enabled here in case you want to try it, but it's pretty ugly! There are probably ways to neaten it up (e.g. create a fork of RN/Hermes), but as this has already been fixed in Hermes master and so should be fixed in an upcoming RN version, I don't think it makes sense for us to try to officially support a workaround right now.

Essentially the steps are:

1. Take Hermes 0.9.0 (used in the current RN version) and pull in https://github.com/facebook/hermes/commit/2a32afb225349c75f3aeb4ffca10f3c4daab6fad to allow disabling the timer
2. Recompile Hermes from source
3. Drop this compiled Hermes into your RN project, overwriting the existing one

In more detail – I figured these steps out from their CircleCI workflow and have reconstructed the steps from memory/my shell history, so apologies if I miss something, let me know if you get stuck.

For iOS:

1. Clone my fork of the Hermes repo from https://github.com/tomduncalf/hermes
2. Check out the td/timer-disabled branch, where I've cherry picked the commit from Hermes which disables the timer onto the v0.9.0 tag (which is what React Native needs)
3. In the hermes clone directory, run ./utils/build-ios-framework.sh and wait ☕
4. Copy the compiled xcframework from hermes/destroot/Library/Frameworks/Universal/hermes.xcframework to your project's ios/Pods/hermes-engine/destroot/Library/Frameworks/universal/hermes.xcframework, overwriting the existing one

For Android:

1. Clone my fork of the Hermes repo from https://github.com/tomduncalf/hermes
2. Check out the td/timer-disabled branch, where I've cherry picked the commit from Hermes which disables the timer onto the v0.9.0 tag (which is what React Native needs)
3. Create a directory at the same level as your hermes clone called hermes_ws and cd hermes_ws
4. Set an env var pointing to this dir:

   export HERMES_WS_DIR=`pwd`
   

5. Create a symlink to hermes inside hermes_ws: ln -s ../hermes
6. Configure the hermes compiler build: hermes/utils/build/configure.py ./build
7. Build the hermes compiler: cmake --build ./build --target hermesc
8. Build the hermes package:

   export ANDROID_SDK="$ANDROID_HOME"
   export ANDROID_NDK="$ANDROID_HOME/ndk-bundle"
   cd "$HERMES_WS_DIR/hermes/android" && ./gradlew githubRelease
   

9. Unzip the build artefact, hermes_ws/build_android/distributions/hermes-runtime-android-v0.9.0.tar.gz
10. In your React Native project directory, overwrite the files in node_modules/hermes-engine/android/ with the files you unzipped in step 9.

[mfbx9da4]

Thanks for doing so though in my case Hermes is not enabled.

[tomduncalf]

Ah OK, we don't have a workaround for that case yet. I'll update this issue when we have a fix though

[tradebulls]

@tomduncalf A new update to React Native is available now. https://github.com/facebook/react-native/blob/main/CHANGELOG.md

Are the Hermes performance issue fixed in this RN release? Can we expect an update to realm with performance issue fixes with Hermes Enabled?

I guess with new update, the cost for JSI abstraction layer might be reduced. Thanks in advance

[tomduncalf]

Hey @tradebulls, thanks for letting me know – I'll take a look at the new release and see. They mention they are now building Hermes from source so hopefully this will pull in the recent changes in Hermes. I'll update you once I've had a chance to look.

[tradebulls]

Thanks for doing so. Looking forward to your reply

[tomduncalf]

It looks like this will require some work from our side to support, if I drop the new React Native into our Hermes branch I get a Hermes error. We will update you once we have some progress on this, it is a priority for us as we know these performance regressions are causing issues.

[tomduncalf]

Hey @tradebulls, unfortunately I'm still seeing a performance regression with the latest RN. I'm following up with Meta to see what we can do mitigate this.

[tradebulls]

@tomduncalf Thanks for doing so.

[tomduncalf]

I've confirmed that this change did not make it to RN 0.69, but should be in RN 0.70 which should hopefully be out fairly soon.

[tradebulls]

Ohhk...to have a new major version of RN to be released takes about 2+months. As RN0.69 is recently rolled out, they would be providing with minor version release for a while(Eg Rn 0.69.1 or RN 0.69.2). @tomduncalf any idea if they are planning to put it in minor release as it would be out soon...

[tomduncalf]

Obviously I can't make any commitments for the Meta team, but it sounded like they were hoping to get this out sooner than that

[tradebulls]

Ok.. Thanks for the update :)

[tradebulls]

@tomduncalf is the performance issue address in Rn 0.69.1? Something similar to performance fixes by RN, not really issue if this is related to our performance issue. Use monotonic clock for performance.now() (114d31feee)

[tomduncalf]

@tradebulls I think that's something different. The issue causing the regression is this timer code in Hermes which times every C++ <-> JS call, which results in a lot of extra CPU usage for Realm as we make a lot of these calls. There is a compile-time option to disable the timer (HERMESJSI_DISABLE_STATS_TIMER) but the current build of Hermes in React Native 0.69 did not have this flag enabled when they compiled it. We're expecting it to be fixed in the first RC of 0.70.

[tradebulls]

Ok.. Thanks for the clarity...Really appreciated :)

[mfbx9da4]

Hey, any timeline on the non-Hermes situation? When can we expect an update about how long it will take to resolve?

[tradebulls]

@mfbx9da4 It seems RN 0.70.0 RC is out with the changelog. However no updates on HERMESJSI_DISABLE_STATS_TIMER disabled feature as of now. You can check the same here - https://github.com/facebook/react-native/releases/tag/v0.70.0-rc.0