ArcadeData
Speed degradation over time (which can means 15 minutes or less if intense usage)
Hello,
On the 14th of april at the office we launched the RC1 of our application. More like a "hands on" were people could connect to the tool and use it. Long story short it's a rag app, so each query requires muliples Cypher/SQL arcadedb queries.
We ended up meeting slow downs on the application at first we were not sure on which part of the pipeline we had slow down, could be the embeddings, reranker, LLM, but some parts of the applications that were just calling a server endpoint (python flask server) that itself was just calling the rest API was very slow to answer, like a command to list databases could take more time that it should, (>0.4s).
- Did you already meet this kind of issue and could it just be a configuration fault ?
- Using default config on a server-desktop machine, version 25.3.2
- Specs are ubuntu 22, 128gb ram and powerful Ryzen 7 CPU with 12 cores (28 threads)
- If it's something you didn't meet before what could we do you help you reproduce the error/get some logs ?
We were kinda busy and didn't really met this issue after because we went back to dev usage with one concurrent user. But now we're having another "hands on" in Poland and we just met this issue 24h before the event.
Thanks.
Doubt it, unless after the INSERTION operations are done it still slow downs the READ operations.
I fixed a few performance issues in the past days in the last snapshot. Is there any chance you can test the latest snapshot?
I understand this kind of issue « it happend some time with some X and I cannot reproduce it » Isn’t helping at all, it was more like a distress call asking if someone met this issue.
I could write a script that would simulate multiple users to reproduce on old versions first
Can you provide a Jstack of the JVM? Not here, just send it to support at arcadedata.com . A test to replicate the scenario would be great, of course. Moreover: how is ArcaeDB deployed? Docker, or "bare metal"? Any special configuration for the JVM, like GC? How much memory to the JVM?
It's bare metal, default config for everything, we just added the .sh file that starts the service to system service, nothing more. I'm trying to find time to reproduce the bug
Which JVM and version are you using? Even if 25.3.2 is compiled using java 17, you can take advantages of newest JVMs, for example G1 as garbage collector. How much ram to the database? By default, JVM should use:
- Initial heap size of 1/64 of physical memory
- Maximum heap size of 1/4 of physical memory So, in your case, maximun heap size is 32Gb
Hi, we worked a lot and we ran some llong run load tests using: https://github.com/ArcadeData/arcadedb/tree/main/e2e-perf/src/test/java/com/arcadedb/test/performance
We are going to release, I hope that all these fixes on transactions, lock, index compaction could help
No rush on my end I just started back working on the project at the office, thanks for the support !
Ok, closing this issue for now, please comment to reopen if you have further information we can help with.