ParseResults failing for larger reports

[kstoner-sbux]

Describe the bug For long running tests with a lot of API's (what Amazonians call a "gameday" test) test-results are not being logged in Dynamo and the Test Status message on the dashboard reads:

Test Failed Failed to parse the results.

UPDATE: I am wondering if this is related to the 400KB item size limit in DynamoDB. Since the DLT framework is storing all of a test's history in the same item as the rest of the test's metadata the item size will grow over time. This means that even small tests will eventually hit this bug.

SUGGESTION: The history for a test should be stored in individual test-history items that relate to the parent test in a one-to-many relationship.

To Reproduce In my test I am executing 40+ API's for a combined load of 4k+ TPS for approximately 2 hours. I have attached the offending results.xml file with obfuscated Group names.

Run a test with multiple samplers in the same JMX file over and over again until the test's DynamoDB item size starts to approach 400 KB.3

Expected behavior The method should not fail to parse a properly formatted results file.

Please complete the following information about the solution:

• [ ] Version: 2.0.1
• [ ] Region: us-west-2
• [ ] Was the solution modified from the version published on this repository? Only the container. The Lambda functions are untouched.
• [ ] If the answer to the previous question was yes, are the changes available on GitHub? No - contact me and I can send you a zip file.
• [ ] Have you checked your service quotas for the sevices this solution uses? Yes
• [ ] Were there any errors in the CloudWatch Logs? Yes: 2022-02-16T03:36:01.947Z 31fcd66f-59d1-4f17-a7d2-131042b5bd6d ERROR finalResults function error ValidationException: Item size to update has exceeded the maximum allowed size at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27) at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20) at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10) at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:688:14) at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12) at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9) at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:690:12) at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18) { code: 'ValidationException', time: 2022-02-16T03:36:01.869Z, requestId: 'HSMA0VQVRIRUM6TQ2P4G28R5EBVV4KQNSO5AEMVJF66Q9ASUAAJG', statusCode: 400, retryable: false, retryDelay: 20.95974031511567 }

Screenshots

Additional context

results.xml.txt

[emcfins]

Thank you for submitting this bug. We will look into it and add it to our backlog.

[kstoner-sbux]

UPDATE: This appears to me to be a serious bug that everyone will hit at some point. The more an individual test is run, to quicker that the test will start to fail with the described failure in the ParseResults.

[emcfins]

Thank you for this update. We are looking to address this issue in upcoming releases.

[jmnxno]

I've been hitting this big time. I've tried to work around it, but no luck yet... Any ideas or suggestions would be helpful. Thanks.

[G-Lenz]

We have addressed this in V3.0.0. We have moved the test history into a separate DynamoDB table.

[shanewarnerthrive]

Still seeing this issue on v3.0.0. Ran 10 tasks with 60 concurrent with a 5 minute ramp up and 60 minute hold time. .log output files are ~450KB in size.

[zxkane]

Met the same problem when using v3.0.0. Lots of empty jmeter xml generated in test result bucket.

Found below logs from the out of jmeter task runner, looks like there is no disk space left in fargate runner

Created the tree successfully using /tmp/artifacts/modified_requests.jmx
Starting standalone test @ Mon Oct 24 07:45:19 UTC 2022 (1666597519397)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
2022-10-24 08:06:20,268 simple countly event by Restful API-ThreadStarter 1-9 ERROR Unable to write to stream /tmp/artifacts/jmeter.log for appender jmeter-log org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to stream /tmp/artifacts/jmeter.log
	at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:252)
	at org.apache.logging.log4j.core.appender.FileManager.writeToDestination(FileManager.java:277)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager.java:283)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:294)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:199)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:190)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:181)
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
	at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:542)
	at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:500)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:483)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
	at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82)
	at org.apache.logging.log4j.core.Logger.log(Logger.java:161)
	at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2205)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2159)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2142)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2034)
	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1899)
	at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:184)
	at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:298)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: No space left on device
	at java.base/java.io.FileOutputStream.writeBytes(Native Method)
	at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:250)
	... 24 more

2022-10-24 08:06:20,271 simple countly event by Restful API-ThreadStarter 1-8 ERROR Unable to write to stream /tmp/artifacts/jmeter.log for appender jmeter-log org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to stream /tmp/artifacts/jmeter.log
	at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:252)
	at org.apache.logging.log4j.core.appender.FileManager.writeToDestination(FileManager.java:277)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager.java:283)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:294)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:199)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamA