knuu icon indicating copy to clipboard operation
knuu copied to clipboard
verified publisher icon celestiaorg

migrate chaos mesh to the knuu

Open mojtaba-esk opened this issue 1 year ago • 3 comments

Closes #451

it is in draft mode because the tests are flaky, need to find a way to assure the tests always.

The tests are passing when run one by one, but when run altogether some of them fail. The attempts done to mitigate this behavior:

  1. Running in sequence
  2. Running in individual namespaces (each test has its own knuu object)
  3. Added a function to wait until the Chaos rule is applied
  4. Added some delay time to make sure the Chaos traffic shape rule is applied & effective
  5. Increased the size of the bigfile to download to make the elapsed time measurement more accurate

Even tried to run them separately in different terminals, they also fail when they are running at the same time.

mojtaba-esk avatar Jun 25 '24 16:06 mojtaba-esk

[!IMPORTANT]

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

[!TIP]

Early access features: enabled

We are currently testing the following features in early access:

  • OpenAI gpt-4o model for code reviews and chat: OpenAI claims that this model is better at understanding and generating code than the previous models. We seek your feedback over the next few weeks before making it generally available.

Note:

  • You can enable or disable early access features from the CodeRabbit UI or by updating the CodeRabbit configuration file.
  • Please join our Discord Community to provide feedback and report issues.
  • OSS projects are currently opted into early access features by default.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

coderabbitai[bot] avatar Jun 25 '24 16:06 coderabbitai[bot]

Can you add it to the github workflow matrix so we can see the failures in CI? https://github.com/celestiaorg/knuu/blob/8d1b00fcae71de633bf5236bbec903d4b859303d/.github/workflows/knuu_testing.yml#L14

Can you share what the failures are that you are seeing locally?

MSevey avatar Jun 25 '24 18:06 MSevey

Can you add it to the github workflow matrix so we can see the failures in CI?

done

Can you share what the failures are that you are seeing locally?

Sure. Here is when I run it all together with isolated namespaces sequentially:

=== RUN   TestRunSuite
=== RUN   TestRunSuite/TestBandwidth
time="2024-06-26T10:41:51+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:41:51+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-104151-940
    chaosmesh_test.go:140: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:145: Time taken for wget before bandwidth: 27.345988762s 	 after bandwidth: 12.980991868s
    chaosmesh_test.go:147: 
        	Error Trace:	/home/moji/celestia/knuu/e2e/chaosmesh/chaosmesh_test.go:147
        	Error:      	"12.980991868s" is not greater than "27.345988762s"
        	Test:       	TestRunSuite/TestBandwidth
        	Messages:   	Time after bandwidth is not longer than time before bandwidth
=== RUN   TestRunSuite/TestCorrupt
time="2024-06-26T10:42:56+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:42:56+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-104256-035
    chaosmesh_test.go:108: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:113: Time taken for wget before corrupt: 29.818175866s 	 after corrupt: 14.994630351s
    chaosmesh_test.go:115: 
        	Error Trace:	/home/moji/celestia/knuu/e2e/chaosmesh/chaosmesh_test.go:115
        	Error:      	"14.994630351s" is not greater than "29.818175866s"
        	Test:       	TestRunSuite/TestCorrupt
        	Messages:   	Time after corrupt is not longer than time before corrupt
=== RUN   TestRunSuite/TestDelay
time="2024-06-26T10:44:04+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:44:04+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-104404-580
    chaosmesh_test.go:27: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:32: Time taken for wget before delay: 26.889980033s 	 after delay: 26.170361577s
    chaosmesh_test.go:34: 
        	Error Trace:	/home/moji/celestia/knuu/e2e/chaosmesh/chaosmesh_test.go:34
        	Error:      	"26.170361577s" is not greater than "26.889980033s"
        	Test:       	TestRunSuite/TestDelay
        	Messages:   	Time after delay is not longer than time before delay
=== RUN   TestRunSuite/TestDuplicate
time="2024-06-26T10:45:19+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:45:19+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-104519-042
    chaosmesh_test.go:81: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:86: Time taken for wget before duplicate: 27.027887443s 	 after duplicate: 26.465900458s
    chaosmesh_test.go:88: 
        	Error Trace:	/home/moji/celestia/knuu/e2e/chaosmesh/chaosmesh_test.go:88
        	Error:      	"26.465900458s" is not greater than "27.027887443s"
        	Test:       	TestRunSuite/TestDuplicate
        	Messages:   	Time after duplicate is not longer than time before duplicate
=== RUN   TestRunSuite/TestLoss
time="2024-06-26T10:46:42+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:46:42+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-104642-121
    chaosmesh_test.go:54: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:59: Time taken for wget before loss: 13.95433273s 	 after loss: 14.305350963s
--- FAIL: TestRunSuite (343.31s)
    --- FAIL: TestRunSuite/TestBandwidth (64.09s)
    --- FAIL: TestRunSuite/TestCorrupt (68.55s)
    --- FAIL: TestRunSuite/TestDelay (74.46s)
    --- FAIL: TestRunSuite/TestDuplicate (83.08s)
    --- PASS: TestRunSuite/TestLoss (53.13s)
FAIL
FAIL	github.com/celestiaorg/knuu/e2e/chaosmesh	343.448s
FAIL

and when we run them one by one, they usually pass:

go test -v ./e2e/chaosmesh/ -timeout 60m -count=1 --run TestRunSuite/TestBandwidth
go test -v ./e2e/chaosmesh/ -timeout 60m -count=1 --run TestRunSuite/TestCorrupt
go test -v ./e2e/chaosmesh/ -timeout 60m -count=1 --run TestRunSuite/TestDelay
go test -v ./e2e/chaosmesh/ -timeout 60m -count=1 --run TestRunSuite/TestDuplicate
go test -v ./e2e/chaosmesh/ -timeout 60m -count=1 --run TestRunSuite/TestLoss
=== RUN   TestRunSuite
=== RUN   TestRunSuite/TestBandwidth
time="2024-06-26T10:54:40+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:54:40+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-105440-946
    chaosmesh_test.go:140: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:145: Time taken for wget before bandwidth: 13.117046776s 	 after bandwidth: 27.103344713s
--- PASS: TestRunSuite (65.91s)
    --- PASS: TestRunSuite/TestBandwidth (65.91s)
PASS
ok  	github.com/celestiaorg/knuu/e2e/chaosmesh	66.045s
=== RUN   TestRunSuite
=== RUN   TestRunSuite/TestCorrupt
time="2024-06-26T10:55:49+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:55:49+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-105549-726
    chaosmesh_test.go:108: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:113: Time taken for wget before corrupt: 26.214538065s 	 after corrupt: 26.921705398s
--- PASS: TestRunSuite (77.70s)
    --- PASS: TestRunSuite/TestCorrupt (77.70s)
PASS
ok  	github.com/celestiaorg/knuu/e2e/chaosmesh	77.845s
=== RUN   TestRunSuite
=== RUN   TestRunSuite/TestDelay
time="2024-06-26T10:57:10+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:57:10+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-105710-157
    chaosmesh_test.go:27: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:32: Time taken for wget before delay: 28.403200834s 	 after delay: 29.653790723s
--- PASS: TestRunSuite (83.11s)
    --- PASS: TestRunSuite/TestDelay (83.11s)
PASS
ok  	github.com/celestiaorg/knuu/e2e/chaosmesh	83.239s
=== RUN   TestRunSuite
=== RUN   TestRunSuite/TestDuplicate
time="2024-06-26T10:58:36+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:58:36+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-105836-117
    chaosmesh_test.go:81: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:86: Time taken for wget before duplicate: 14.752317089s 	 after duplicate: 13.546388261s
    chaosmesh_test.go:88: 
        	Error Trace:	/home/moji/celestia/knuu/e2e/chaosmesh/chaosmesh_test.go:88
        	Error:      	"13.546388261s" is not greater than "14.752317089s"
        	Test:       	TestRunSuite/TestDuplicate
        	Messages:   	Time after duplicate is not longer than time before duplicate
--- FAIL: TestRunSuite (52.02s)
    --- FAIL: TestRunSuite/TestDuplicate (52.02s)
FAIL
FAIL	github.com/celestiaorg/knuu/e2e/chaosmesh	52.155s
FAIL
=== RUN   TestRunSuite
=== RUN   TestRunSuite/TestLoss
time="2024-06-26T10:59:30+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:59:30+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-105930-866
    chaosmesh_test.go:54: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:59: Time taken for wget before loss: 14.224850495s 	 after loss: 26.397818292s
--- PASS: TestRunSuite (64.47s)
    --- PASS: TestRunSuite/TestLoss (64.47s)
PASS
ok  	github.com/celestiaorg/knuu/e2e/chaosmesh	64.594s

mojtaba-esk avatar Jun 26 '24 09:06 mojtaba-esk

Can we close this PR?

smuu avatar Sep 18 '24 07:09 smuu

Can we close this PR?

Well if we decide not to use Chaos mesh in knuu, then we can close it.

mojtaba-esk avatar Sep 18 '24 07:09 mojtaba-esk

Can we close this PR?

Well if we decide not to use Chaos mesh in knuu, then we can close it.

The teams do not use knuu with network shaping and it seems they will not in the near future.

smuu avatar Sep 19 '24 08:09 smuu