Request body larger than maxBodyLength limit

[Baddooo14]

Issue description

At seemingly random intervals my node seems to receive the following error: ERROR: Failed to write into Blazegraph: Error [ERR_FR_MAX_BODY_LENGTH_EXCEEDED]: Request body larger than maxBodyLength limit This is an axios issue, where the data that is being submitted is too large too handle. The variable maxBodyLength can be set to Infinity, in order to allow larger datasets to be published, but I couldn't find the config file while looking through the codebase. I also don't know if you'd want that.

I feel like this has to do with publishing data, and the telemetry error that accompanies the error is TripleStoreInsertError.

I was running the script supplied by Aescwine, which spams the node with the same operations over and over again. It also publishes the exact same data over and over again. Therefore I do no believe it has something to do with the actual publishing of the data that I am trying to send.

I believe it is the telemetry submission which has gotten too large to be published. This also corroborates the story that we broke the telemetry system by spamming the nodes with operations. The hourly telemetry submissions will then perhaps fail and no score gets awarded to nodes receiving this error.

This might already be fixed by the rate limiting implemented in 1.30, as I have not seen the error since. I also changed the telemetry code yesterday (won't do it anymore), to submit telemetry data every 5 minutes. Since that moment I have not received the above errors anymore.

Expected behavior

No error

Actual behavior

Described above

Steps to reproduce the problem

1. Run Aescwine script (old version which was spamming operations)
2. Wait for the error to appear

Specifications

• Node version: V6 1.29
• Platform: Ubuntu 20.04
• Node wallet: 0xa705c6A78d99001383C39946225B0E22630A0BdF
• Node libp2p identity: QmbeMcpn6JuubZGgFSJfy5X4FshcEAJmh8RafmL3bmMpEb

Contact details

• Email: [email protected]

Error logs

[2022-03-18 06:56:52] ERROR: Failed to write into Blazegraph: Error [ERR_FR_MAX_BODY_LENGTH_EXCEEDED]: Request body larger than maxBodyLength limit - Error [ERR_FR_MAX_BODY_LENGTH_EXCEEDED]: Request body larger than maxBodyLength limit [2022-03-18 06:56:52] USERLVL: Found error will be reported to Telemetry: TripleStoreInsertError

Disclaimer

Please be aware that the issue reported on a public repository allows everyone to see your node logs, node details, and contact details. If you have any sensitive information, feel free to share it by sending an email to [email protected].

[Baddooo14]

Update:

I still seem to get the errors, albeit less than before.

So it's definitely not fixed by 1.30 rate limiting. I don't know, what's causing it. In my logs it doesn't happen after a specific type of operation every time, but it must be related to some kind of publishing.

[Valcyclovir]

Can you provide more details about your system ?

Also provide logs on Blazegraph and make sure it is running. journalctl -u blazegraph --output cat -fn 100

I have not seen anyone else with this problem so far. Check blazegraph logs and check if you really fixed the telemetry publishing back to hourly. This could be due to Aescwine's publishing script and not the dkgv6 at all.

[Baddooo14]

System:

Operating System | Ubuntu Server 20.04 Disk Space | 60 GB Memory | 4 GB CPU Cores | 3 CPU(s)

Blazegraph looked fine:

Started Blazegraph - OriginTrail V6 Stage 1 Beta Node.
INFO: com.bigdata.util.config.LogUtil: Configure: jar:file:/root/blazegraph.jar!/log4j.properties
BlazeGraph(TM) Graph Engine
                   Flexible
                   Reliable
                  Affordable
      Web-Scale Computing for the Enterprise
Copyright SYSTAP, LLC DBA Blazegraph 2006-2016.  All rights reserved.
davidnode
Wed Mar 23 07:50:09 UTC 2022
Linux/5.4.0-104-generic amd64
AMD EPYC 7542 32-Core Processor Family 23 Model 49 Stepping 0, AuthenticAMD #CPU=3
Ubuntu 11.0.14
freeMemory=52599360
buildVersion=2.1.6-SNAPSHOT
gitCommit=6b0c935523f5064b80279b30a5175a858cddd2a1
Dependency         License
ICU                http://source.icu-project.org/repos/icu/icu/trunk/license.html
bigdata-ganglia    http://www.apache.org/licenses/LICENSE-2.0.html
blueprints-core    https://github.com/tinkerpop/blueprints/blob/master/LICENSE.txt
colt               http://acs.lbl.gov/software/colt/license.html
commons-codec      http://www.apache.org/licenses/LICENSE-2.0.html
commons-fileupload http://www.apache.org/licenses/LICENSE-2.0.html
commons-io         http://www.apache.org/licenses/LICENSE-2.0.html
commons-logging    http://www.apache.org/licenses/LICENSE-2.0.html
dsiutils           http://www.gnu.org/licenses/lgpl-2.1.html
fastutil           http://www.apache.org/licenses/LICENSE-2.0.html
flot               http://www.opensource.org/licenses/mit-license.php
high-scale-lib     http://creativecommons.org/licenses/publicdomain
httpclient         http://www.apache.org/licenses/LICENSE-2.0.html
httpclient-cache   http://www.apache.org/licenses/LICENSE-2.0.html
httpcore           http://www.apache.org/licenses/LICENSE-2.0.html
httpmime           http://www.apache.org/licenses/LICENSE-2.0.html
jackson-core       http://www.apache.org/licenses/LICENSE-2.0.html
jetty              http://www.apache.org/licenses/LICENSE-2.0.html
jquery             https://github.com/jquery/jquery/blob/master/MIT-LICENSE.txt
jsonld             https://raw.githubusercontent.com/jsonld-java/jsonld-java/master/LICENCE
log4j              http://www.apache.org/licenses/LICENSE-2.0.html
lucene             http://www.apache.org/licenses/LICENSE-2.0.html
nanohttp           http://elonen.iki.fi/code/nanohttpd/#license
rexster-core       https://github.com/tinkerpop/rexster/blob/master/LICENSE.txt
river              http://www.apache.org/licenses/LICENSE-2.0.html
semargl            https://github.com/levkhomich/semargl/blob/master/LICENSE
servlet-api        http://www.apache.org/licenses/LICENSE-2.0.html
sesame             http://www.openrdf.org/download.jsp
slf4j              http://www.slf4j.org/license.html
zookeeper          http://www.apache.org/licenses/LICENSE-2.0.html
WARN : NanoSparqlServer.java:517: Starting NSS
WARN : ServiceProviderHook.java:171: Running.
serviceURL: http://83.138.55.14:9999
Welcome to the Blazegraph(tm) Database.
Go to http://83.138.55.14:9999/blazegraph/ to get started.

I am sure it is back to one hour period: 60 * 60 * 1000, // 1 hour

I have seen someone else on the discord with the same error. NZT advised us to open an issue. I don't know whether that person was running the same script.

I must admit that I have only seen 1 new error the past 2 days, while before that it was around 4 times per day. (Though I was publishing around 10x times as much those previous days).

[Baddooo14]

Here you can see other people having this issue.

[Valcyclovir]

were you all running the same publish script ?

[Baddooo14]

I have no idea. At least me and aescwine were, no idea about the other guy. However the script now very much conforms to the rate limits etc, and I still got one of those errors yesterday.

[Valcyclovir]

Very intriguing. Would be interesting to see whether this error occurs with other scripts made by the community, like the ones from cosmicloud here and here or from sam here. I personally tried them all, have a 4-5gb journal, did a grep and did not find any of your error. The main difference on aescwine script is that it spams the same thing over and over, something could've gone wrong there. We can try different things if you bring this over to node chan, perhaps see if you get the same error with the other publishing scripts. Cheers !

[Aescwine]

Hi @Valcyclovir, I've seen 3 of these errors in my logs, all on the 18th March. It's not obvious to me if it's occurring with data I'm publishing or that I'm storing for other nodes.

As @Baddooo14 mentions, the telemetry problems seemed to begin on the day I shared by script. My script is a loop that provisions an asset, and then makes a number of request across all of the APIs: resolving, searching (entity and assertion), sparql query and proofs.

Initially it was sending thousands of requests per hour i.e 1 publish request, wait 30 seconds, then send 500 resolve requests at 750ms intervals, 200 search requests at 1 second intervals and so on. Seeing as the telemetry process runs hourly to publish the event data and with so many requests, it crossed my mind that we might be hitting the limit due there being too much telemetry data to publish.

I had also lowered the telemetry process period to 5 minutes to see if my requests were being picked up in the dashboard sooner, which is probably why I'd not seen more of these errors, with the telemetry publish requests containing a lot less data.

[drMurlly]

I have an ERROR again on one of my test nodes: Error in send telemetry command: Error [ERR_FR_MAX_BODY_LENGTH_EXCEEDED]:

I had the same Error in one of the earlier test versions. I don't know exactly which version. The Error appears just on one of my test nodes, and other nodes are not affected. Therefore node is not able to send telemetry data.

[NZT48]

This issue is being closed as inactive due to the date of the last activity on it. If you believe this is still a problem, please create a new issue and confirm that it is reproducible in the current ot-node release version. We are working towards closing open issues that meet specific criteria and ask you to create a new one for those that that are truly bugs in current release. We'll be monitoring those issues so that they are properly managed.

Thank you, OriginTrail Team