react-native-reanimated icon indicating copy to clipboard operation
react-native-reanimated copied to clipboard

App crashes on reload in development mode with JSC Engine

Open ykliuiev opened this issue 11 months ago • 49 comments

Description

After upgrading to react-native-reanimated v3, we noticed that app crashes on reload in development mode with JSC engine. It's reproducible in Example/ app in react-native-reanimated repo and can confirm it does not happen with hermes engine enabled.

Steps to reproduce

  1. Disable hermes in Example/ios/Podfile file by changing :hermes_enabled => flags[:hermes_enabled], :fabric_enabled => flags[:fabric_enabled], to :hermes_enabled => false, :fabric_enabled => false,
  2. reinstall pods by running pod install command from Example/ios folder
  3. build the app and launch it
  4. reload the app with R or devmode

Thank you :raised:

Snack or a link to a repository

https://github.com/ykliuiev/reanimated-crash-repro

Reanimated version

3.3.0

React Native version

0.72.3

Platforms

iOS

JavaScript runtime

JSC

Workflow

React Native (without Expo)

Architecture

Paper (Old Architecture)

Build type

Debug mode

Device

iOS simulator

Device model

No response

Acknowledgements

Yes

ykliuiev avatar Jul 20 '23 12:07 ykliuiev

Same here #4778

victor-asdf avatar Jul 20 '23 12:07 victor-asdf

@victor-asdf yes, looks like the same but we identified that the root cause is JSC runtime which should help narrow down the potential scope for debugging and investigation.

ykliuiev avatar Jul 20 '23 13:07 ykliuiev

@ykliuiev Thanks. I have tried all versions of 3.x and it seems happen all the time

victor-asdf avatar Jul 20 '23 13:07 victor-asdf

@tomekzaw cc

TheMakerOleguch avatar Jul 20 '23 14:07 TheMakerOleguch

iOS App is crashing in iOS 16.5.1 & above in during first launch in Real Device. Not found in the older iOS version.

When: During first launch of the App after installation. Second launch onwards, no crash. It is an intermittent crash and does not happen every first time launch of the app after installation.

OS version: iOS 16.5.1 & above. Not seen in below version react-native-reanimated version: 3.1.0 React Native version: 0.71.7 React Native Type: React Native CLI based app. iPhone: Reproduced in iPhone 13 Pro Max, iPhone 14 Pro Max, iPhone 12, iPhone XR

Podfile changes: Below Podfile changes done to support react-native-firebase library

use_frameworks! :linkage => :static // use static frameworks :hermes_enabled => flags[:hermes_enabled], // No change :fabric_enabled => false // fabric disabled #:flipper_configuration => flipper_config, // filpper commented

Crash Log:

Crashed: com.facebook.react.JavaScript 0 xy_app 0x1bd800 reanimated::ShareableRemoteFunction::~ShareableRemoteFunction() + 409 (jsi.h:409) 1 xy_app 0x1b9b98 std::__1::shared_ptrreanimated::ShareableRemoteFunction::~shared_ptrabi:v15006 + 216 (shared_ptr.h:216) 2 xy_app 0x1baf38 std::__1::__function::__func<reanimated::NativeReanimatedModule::NativeReanimatedModule(std::__1::shared_ptrfacebook::react::CallInvoker const&, std::__1::shared_ptrreanimated::Scheduler const&, std::__1::shared_ptrfacebook::jsi::Runtime const&, std::__1::shared_ptrreanimated::ErrorHandler const&, std::__1::function<facebook::jsi::Value (facebook::jsi::Runtime&, int, facebook::jsi::String const&)>, reanimated::PlatformDepMethodsHolder)::$_1::operator()(facebook::jsi::Runtime&, facebook::jsi::Value const&, facebook::jsi::Value const&) const::'lambda'(), std::__1::allocator<reanimated::NativeReanimatedModule::NativeReanimatedModule(std::__1::shared_ptrfacebook::react::CallInvoker const&, std::__1::shared_ptrreanimated::Scheduler const&, std::__1::shared_ptrfacebook::jsi::Runtime const&, std::__1::shared_ptrreanimated::ErrorHandler const&, std::__1::function<facebook::jsi::Value (facebook::jsi::Runtime&, int, facebook::jsi::String const&)>, reanimated::PlatformDepMethodsHolder)::$_1::operator()(facebook::jsi::Runtime&, facebook::jsi::Value const&, facebook::jsi::Value const&) const::'lambda'()>, void ()>::destroy_deallocate() + 256 (new:256) 3 xy_app 0x279d94 std::__1::__function::__func<facebook::react::Instance::JSCallInvoker::scheduleAsync(std::__1::function<void ()>&&)::$_3, std::__1::allocator<facebook::react::Instance::JSCallInvoker::scheduleAsync(std::__1::function<void ()>&&)::$_3>, void (facebook::react::JSExecutor*)>::destroy_deallocate() + 256 (new:256) 4 xy_app 0x283f50 std::__1::__function::__alloc_func<facebook::react::NativeToJsBridge::runOnExecutorQueue(std::__1::function<void (facebook::react::JSExecutor*)>)::$_8, std::__1::allocator<facebook::react::NativeToJsBridge::runOnExecutorQueue(std::__1::function<void (facebook::react::JSExecutor*)>)::$_8>, void ()>::destroyabi:v15006 + 300 (NativeToJsBridge.cpp:300) 5 xy_app 0x283d9c std::__1::__function::__func<facebook::react::NativeToJsBridge::runOnExecutorQueue(std::__1::function<void (facebook::react::JSExecutor*)>)::$_8, std::__1::allocator<facebook::react::NativeToJsBridge::runOnExecutorQueue(std::__1::function<void (facebook::react::JSExecutor*)>)::$_8>, void ()>::destroy_deallocate() + 256 (new:256) 6 xy_app 0x218830 std::__1::shared_ptr<std::__1::function<void ()>>::~shared_ptrabi:v15006 + 216 (shared_ptr.h:216) 7 xy_app 0x218c4c std::__1::__function::__func<facebook::react::RCTMessageThread::runOnQueue(std::__1::function<void ()>&&)::$_1, std::__1::allocator<facebook::react::RCTMessageThread::runOnQueue(std::__1::function<void ()>&&)::$_1>, void ()>::destroy_deallocate() + 700 (shared_ptr.h:700) 8 libsystem_blocks.dylib 0x2134 _call_dispose_helpers_excp + 48 9 libsystem_blocks.dylib 0x1d64 _Block_release + 252 10 CoreFoundation 0xaa238 __CFRunLoopDoBlocks + 404 11 CoreFoundation 0x7a0e8 __CFRunLoopRun + 848 12 CoreFoundation 0x7f3ec CFRunLoopRunSpecific + 612 13 xy_app 0x201fb4 +[RCTCxxBridge runRunLoop] + 336 (RCTCxxBridge.mm:336) 14 Foundation 0x5b544 NSThread__start + 716 15 libsystem_pthread.dylib 0x16b8 _pthread_start + 148xy_app 16 libsystem_pthread.dylib 0xb88 thread_start + 8

rajrao1 avatar Jul 23 '23 06:07 rajrao1

Same here! The app crashes whenever we reload the app in development mode and when hermes = false :(

react-native: ^0.72.3
react-native-reanimated: ^3.3.0

Audrey-Ann avatar Jul 24 '23 18:07 Audrey-Ann

added simple repro just in case https://github.com/ykliuiev/reanimated-crash-repro

ykliuiev avatar Jul 25 '23 20:07 ykliuiev

Same here 1464

maydersonmellops avatar Aug 02 '23 17:08 maydersonmellops

Crash also happens (ios only) when doing a codepush update in the production app since it reloads the app in order to apply a new bundle. Would be great to at least get a workaround or a patch. cc @piaskowyk

DaniyarJakupov avatar Aug 07 '23 07:08 DaniyarJakupov

We will check it, but please give us a bit of time 🙏

piaskowyk avatar Aug 07 '23 08:08 piaskowyk

I'm also facing the same issue. Please kindly assist us by performing a quick check. Thank you.

dennytosp avatar Aug 07 '23 08:08 dennytosp

@piaskowyk thank you! The whole stack trace on sentry is huge, but here is some relevant part I think:

Exception Type: EXC_CRASH (SIGABRT)
Crashed Thread: 2

Application Specific Information:
~JSCRuntime > Assertion failed: (objectCounter_ == 0 && "JSCRuntime destroyed with a dangling API object"), function ~JSCRuntime, file JSCRuntime.cpp, line 414.
 > JSCRuntime.cpp > objectCounter_ == 0 && "JSCRuntime destroyed with a dangling API object"

Thread 2 Crashed:
0   libsystem_kernel.dylib          0x3706bd964         __pthread_kill
1   libsystem_pthread.dylib         0x3e2ff3374         pthread_kill
2   libsystem_c.dylib               0x3178a1f4c         abort
3   libsystem_c.dylib               0x3178f8968         __assert_rtn
4   dev                    0x2006bc5ac         facebook::jsc::JSCRuntime::~JSCRuntime
5   dev                    0x2006bc618         [inlined] facebook::jsc::JSCRuntime::~JSCRuntime (JSCRuntime.cpp:403)
6   dev                    0x2006bc618         facebook::jsc::JSCRuntime::~JSCRuntime (JSCRuntime.cpp:403)
7   dev                    0x2005deda4         [inlined] std::__1::__shared_count::__release_shared[abi:v15006] (shared_ptr.h:174)
8   dev                    0x2005deda4         [inlined] std::__1::__shared_weak_count::__release_shared[abi:v15006] (shared_ptr.h:215)
9   dev                    0x2005deda4         [inlined] std::__1::shared_ptr<T>::~shared_ptr[abi:v15006] (shared_ptr.h:702)
10  dev                    0x2005deda4         [inlined] std::__1::shared_ptr<T>::~shared_ptr[abi:v15006] (shared_ptr.h:700)
11  dev                    0x2005deda4         std::__1::shared_ptr<T>::reset[abi:v15006] (shared_ptr.h:768)
12  dev                    0x2005deb34         reanimated::NativeReanimatedModule::~NativeReanimatedModule (NativeReanimatedModule.cpp:202)
13  dev                    0x2005ed8a4         [inlined] std::__1::__shared_count::__release_shared[abi:v15006] (shared_ptr.h:174)
14  dev                    0x2005ed8a4         [inlined] std::__1::__shared_weak_count::__release_shared[abi:v15006] (shared_ptr.h:215)
15  dev                    0x2005ed8a4         std::__1::shared_ptr<T>::~shared_ptr[abi:v15006] (shared_ptr.h:702)
16  dev                    0x2006bf1f8         [inlined] std::__1::shared_ptr<T>::~shared_ptr[abi:v15006] (shared_ptr.h:700)
17  dev                    0x2006bf1f8         [inlined] facebook::jsc::detail::HostObjectProxyBase::~HostObjectProxyBase (JSCRuntime.cpp:751)
18  dev                    0x2006bf1f8         [inlined] facebook::jsc::JSCRuntime::createObject::HostObjectProxy::~HostObjectProxy (JSCRuntime.cpp:768)
19  dev                    0x2006bf1f8         [inlined] facebook::jsc::JSCRuntime::createObject::HostObjectProxy::~HostObjectProxy (JSCRuntime.cpp:768)
20  dev                    0x2006bf1f8         facebook::jsc::JSCRuntime::createObject::HostObjectProxy::finalize (JSCRuntime.cpp:873)
21  JavaScriptCore                  0x317d8ee18         JSC::JSCallbackObject<T>::destroy
22  JavaScriptCore                  0x3183231d0         JSC::MarkedBlock::Handle::specializedSweep<T>
23  JavaScriptCore                  0x31831fba0         JSC::MarkedBlock::Handle::finishSweepKnowingHeapCellType<T>
24  JavaScriptCore                  0x318326c8c         JSC::MarkedBlock::Handle::sweep
25  JavaScriptCore                  0x3182f8d98         JSC::BlockDirectory::lastChanceToFinalize
26  JavaScriptCore                  0x318328ed8         JSC::MarkedSpace::lastChanceToFinalize
27  JavaScriptCore                  0x3182ff514         JSC::Heap::lastChanceToFinalize
28  JavaScriptCore                  0x3188e443c         JSC::VM::~VM
29  JavaScriptCore                  0x3187ac6c8         JSC::JSLockHolder::~JSLockHolder
30  JavaScriptCore                  0x317da0d7c         JSGlobalContextRelease
31  dev                    0x2006bc544         facebook::jsc::JSCRuntime::~JSCRuntime (JSCRuntime.cpp:411)
32  dev                    0x2006bc618         [inlined] facebook::jsc::JSCRuntime::~JSCRuntime (JSCRuntime.cpp:403)
33  dev                    0x2006bc618         facebook::jsc::JSCRuntime::~JSCRuntime (JSCRuntime.cpp:403)
34  dev                    0x2005d9f64         [inlined] std::__1::__shared_count::__release_shared[abi:v15006] (shared_ptr.h:174)
35  dev                    0x2005d9f64         [inlined] std::__1::__shared_weak_count::__release_shared[abi:v15006] (shared_ptr.h:215)
36  dev                    0x2005d9f64         std::__1::shared_ptr<T>::~shared_ptr[abi:v15006] (shared_ptr.h:702)
37  dev                    0x20079887c         [inlined] std::__1::shared_ptr<T>::~shared_ptr[abi:v15006] (shared_ptr.h:700)
38  dev                    0x20079887c         facebook::react::JSIExecutor::~JSIExecutor (JSIExecutor.h:72)
39  dev                    0x200798278         [inlined] facebook::react::JSIExecutor::~JSIExecutor (JSIExecutor.h:72)
40  dev                    0x200798278         facebook::react::JSIExecutor::~JSIExecutor (JSIExecutor.h:72)
41  dev                    0x200656180         [inlined] std::__1::__function::__value_func<T>::operator()[abi:v15006] (function.h:512)
42  dev                    0x200656180         [inlined] std::__1::function<T>::operator() (function.h:1197)
43  dev                    0x200656180         facebook::react::tryAndReturnError (RCTCxxUtils.mm:74)
44  dev                    0x2006612d8         facebook::react::RCTMessageThread::tryFunc (RCTMessageThread.mm:69)
45  dev                    0x2006615a0         facebook::react::RCTMessageThread::runOnQueueSync (RCTMessageThread.mm:92)
46  dev                    0x200776144         facebook::react::NativeToJsBridge::destroy (NativeToJsBridge.cpp:285)
47  dev                    0x20076c2e8         facebook::react::Instance::~Instance (Instance.cpp:39)
48  dev                    0x2006531ec         [inlined] std::__1::default_delete<T>::operator()[abi:v15006] (unique_ptr.h:48)
49  dev                    0x2006531ec         std::__1::__shared_ptr_pointer<T>::__on_zero_shared (shared_ptr.h:263)
50  dev                    0x20064fb84         [inlined] std::__1::__shared_count::__release_shared[abi:v15006] (shared_ptr.h:174)
51  dev                    0x20064fb84         [inlined] std::__1::__shared_weak_count::__release_shared[abi:v15006] (shared_ptr.h:215)
52  dev                    0x20064fb84         [inlined] std::__1::shared_ptr<T>::~shared_ptr[abi:v15006] (shared_ptr.h:702)
53  dev                    0x20064fb84         [inlined] std::__1::shared_ptr<T>::~shared_ptr[abi:v15006] (shared_ptr.h:700)
54  dev                    0x20064fb84         std::__1::shared_ptr<T>::reset[abi:v15006] (shared_ptr.h:768)
55  dev                    0x200650448         __26-[RCTCxxBridge invalidate]_block_invoke (RCTCxxBridge.mm:1309)
56  dev                    0x200656180         [inlined] std::__1::__function::__value_func<T>::operator()[abi:v15006] (function.h:512)
57  dev                    0x200656180         [inlined] std::__1::function<T>::operator() (function.h:1197)
58  dev                    0x200656180         facebook::react::tryAndReturnError (RCTCxxUtils.mm:74)
59  dev                    0x20064bca8         -[RCTCxxBridge _tryAndHandleError:] (RCTCxxBridge.mm:346)
60  Foundation                      0x304a8b454         __NSThreadPerformPerform
61  CoreFoundation                  0x301ac30cc         __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__
62  CoreFoundation                  0x301ad3d8c         __CFRunLoopDoSource0
63  CoreFoundation                  0x301a0e094         __CFRunLoopDoSources0
64  CoreFoundation                  0x301a138a0         __CFRunLoopRun
65  CoreFoundation                  0x301a27464         CFRunLoopRunSpecific
66  dev                    0x20064bbd8         +[RCTCxxBridge runRunLoop] (RCTCxxBridge.mm:337)
67  Foundation                      0x304ab6408         __NSThread__start__
68  libsystem_pthread.dylib         0x3e2fed9a0         _pthread_start

Thread 0
0   libsystem_kernel.dylib          0x3706b7504         mach_msg_trap
1   libsystem_kernel.dylib          0x3706b7b98         mach_msg
2   CoreFoundation                  0x301a0f734         __CFRunLoopServiceMachPort
3   CoreFoundation                  0x301a13a28         __CFRunLoopRun
4   CoreFoundation                  0x301a27464         CFRunLoopRunSpecific
5   GraphicsServices                0x33915b388         GSEventRunModal
6   UIKitCore                       0x306870084         -[UIApplication _run]
7   UIKitCore                       0x3065ee954         UIApplicationMain
8   dev                    0x200176ae0         main (main.m:8)
9   <unknown>                       0x1010b5aa4         <redacted>

DaniyarJakupov avatar Aug 07 '23 10:08 DaniyarJakupov

@tomekzaw @piaskowyk Please provide us with feedback, as we are experiencing an issue and need to resolve it quickly 🥺

Thank you very much

dennytosp avatar Aug 30 '23 05:08 dennytosp

This issue is a bit more complicated, Tomek is still working on it in free time.

piaskowyk avatar Sep 04 '23 06:09 piaskowyk

same here

JavoxirNazarov avatar Sep 09 '23 07:09 JavoxirNazarov

have anyone find, solution?

JavoxirNazarov avatar Sep 09 '23 07:09 JavoxirNazarov

Here's an update: we investigated the issue and found the root cause of the problem.

Before we come up with a proper fix, here's a workaround:

JSCRuntime.cpp

 JSCRuntime::~JSCRuntime() {
   // On shutting down and cleaning up: when JSC is actually torn down,
   // it calls JSC::Heap::lastChanceToFinalize internally which
   // finalizes anything left over.  But at this point,
   // JSValueUnprotect() can no longer be called.  We use an
   // atomic<bool> to avoid unsafe unprotects happening after shutdown
   // has started.
   ctxInvalid_ = true;
   JSGlobalContextRelease(ctx_);
-#ifndef NDEBUG
-  assert(
-      objectCounter_ == 0 && "JSCRuntime destroyed with a dangling API object");
-  assert(
-      stringCounter_ == 0 && "JSCRuntime destroyed with a dangling API string");
-#endif
}

tomekzaw avatar Sep 11 '23 07:09 tomekzaw

Thank you very much, you are awesome @tomekzaw

dennytosp avatar Sep 12 '23 16:09 dennytosp

hey @tomekzaw do you think that without this patch, could this be fixed by using a timeout and reload the app after certain seconds later. I saw this comment in another issue here which is pretty much the same as this one.

ugurcanatas avatar Sep 12 '23 22:09 ugurcanatas

Here's an update: we investigated the issue and found the root cause of the problem.

Before we come up with a proper fix, here's a workaround:

JSCRuntime.cpp

 JSCRuntime::~JSCRuntime() {
   // On shutting down and cleaning up: when JSC is actually torn down,
   // it calls JSC::Heap::lastChanceToFinalize internally which
   // finalizes anything left over.  But at this point,
   // JSValueUnprotect() can no longer be called.  We use an
   // atomic<bool> to avoid unsafe unprotects happening after shutdown
   // has started.
   ctxInvalid_ = true;
   JSGlobalContextRelease(ctx_);
-#ifndef NDEBUG
-  assert(
-      objectCounter_ == 0 && "JSCRuntime destroyed with a dangling API object");
-  assert(
-      stringCounter_ == 0 && "JSCRuntime destroyed with a dangling API string");
-#endif
}

thank you for the patch @tomekzaw,

However for anyone attempting this patch, this caused a crash in production for us. The crash was not consistently happening so it may be hard to catch.

harrisrobin avatar Sep 13 '23 17:09 harrisrobin

I second this, this is a thing. I'll try the suggested workaround meanwhile.

Edit: it does work, but i'm yet to find a way to patch this on the pods postinstall. Any suggestions on how to do it, while we wait for this to be fixed?.

fellenabmb avatar Sep 18 '23 18:09 fellenabmb

Any suggestions on how to do it, while we wait for this to be fixed?.

you aware about patch-package ?

kioltk avatar Sep 23 '23 08:09 kioltk

I second this, this is a thing. I'll try the suggested workaround meanwhile.

Edit: it does work, but i'm yet to find a way to patch this on the pods postinstall. Any suggestions on how to do it, while we wait for this to be fixed?.

@fellenabmb
I hope it helps you. Here is my patch based on @tomekzaw's update.

react-native+0.72.4.patch

diff --git a/node_modules/react-native/ReactCommon/jsc/JSCRuntime.cpp b/node_modules/react-native/ReactCommon/jsc/JSCRuntime.cpp
index 7958519..5ecb903 100644
--- a/node_modules/react-native/ReactCommon/jsc/JSCRuntime.cpp
+++ b/node_modules/react-native/ReactCommon/jsc/JSCRuntime.cpp
@@ -409,12 +409,6 @@ JSCRuntime::~JSCRuntime() {
   // has started.
   ctxInvalid_ = true;
   JSGlobalContextRelease(ctx_);
-#ifndef NDEBUG
-  assert(
-      objectCounter_ == 0 && "JSCRuntime destroyed with a dangling API object");
-  assert(
-      stringCounter_ == 0 && "JSCRuntime destroyed with a dangling API string");
-#endif
 }
 
 std::shared_ptr<const jsi::PreparedJavaScript> JSCRuntime::prepareJavaScript(

dennytosp avatar Sep 24 '23 06:09 dennytosp

@fellenabmb Steps would be:

  1. make the changes to that file in node_modules (node_modules/react-native/ReactCommon/jsc/JSCRuntime.cpp)
  2. run npx patch-package react-native
  3. Add the following to your package.json scripts
"scripts": {
+   "postinstall": "npx patch-package"
 }
  1. now everytime you run npm install or yarn it will also apply your change
  2. Delete your ios/Pods directory
  3. Last, run pod install. The RN package imports files straight from node_modules when installing the pods

Note: if you don't have npx installed you'll need to install patch-package in your project and remove the npx prefix from the commands

If you dont want to actually patch the package (since it's probably not recommended to alter the React Native code) in production, you can make the changes manually locally, and start at step 5 above.

Eli-Nathan avatar Sep 25 '23 08:09 Eli-Nathan

I am not particularly using react-native-reanimated but I am using react-native-bottom-sheet that uses reanimated. My reanimated version is ^3.5.2 and bottom sheet is ^4, I am using JSC. Not Hermes.

I tried to remove the lines inside /react-native/ReactCommon/jsc/JSCRuntime.cpp and rebuilt the app again. Restarted the metro server as well with cache reset. But the app crashes every time when I reload the app in development mode. Is there anything I am doing wrong here? Thanks a lot for any guidance!

Saad-Bashar avatar Oct 25 '23 05:10 Saad-Bashar

the crash also happens on my side with react-native 0.72.6

  1. when metro is reloaded with new js code
  2. when codepush worked

I'm also using JSC not hermes

I confirm that patch listed above: react-native+0.72.4.patch works for react-native 0.72.6

I created a file called patches/react-native+0.72.6.patch and added in package.json this:

"scripts": {
+   "postinstall": "patch-package"
 }

also I installed patch-package yarn add -D patch-package

pavlobu avatar Nov 13 '23 20:11 pavlobu

Here's an update: we investigated the issue and found the root cause of the problem.

@tomekzaw is there an update on the fix? The suggested workaround introduces a potential memory corruption, so running the app is not safe / may have unspecified consequences.

leontiy avatar Nov 15 '23 16:11 leontiy

@leontiy Unfortunately not and this is unlikely to change as we lean towards supporting only Hermes so JSC is not a priority at the moment.

tomekzaw avatar Nov 15 '23 17:11 tomekzaw

I remember we still had issues enabling Hermes for iOS with internationalisation so this isn't great to hear. Will have to see if there's been any progress on getting Hermes to work.

dwxw avatar Nov 16 '23 14:11 dwxw

I think most of the Intl APIs are now working with hermes @dwxw . We are still using JSC for iOS because we see significantly lower memory usage (which is quite surprising). It's been a while since I checked the last time. The whole RN ecosystem is moving towards hermes, so I think what @tomekzaw says makes sense.

michaelknoch avatar Nov 16 '23 14:11 michaelknoch