Load changes gradually when creating a snapshot

[ejolie]

What this PR does / why we need it: We have to load changes gradually from DB when a large number of changes are accumulated.

Which issue(s) this PR fixes:

Fixes #228

Special notes for your reviewer:

I create a draft PR to ask for help. I have two questions while trying to fix the issue. 😅

1. How to determine the right chunk size of changes. Is there any way to test the performance depending on the chunk size of changes in my local environment?

2. How to write test code to make it arrive at the code I changed. I wrote test code referring to other one in TestSnapshot, but I noticed my code does not arrive at 195 line of pushpull.go. I don't know how to set up the situation in the test code.

Thanks in advance.

Does this PR introduce a user-facing change?:

Additional documentation:

Checklist:

• [x] Added relevant tests or not required
• [ ] Didn't break anything

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[hackerwins]

@ejolie Thanks for your contribution. Overall the code looks good.

1. How to determine the right chunk size of changes. Is there any way to test the performance depending on the chunk size of changes in my local environment?

I think we can test the performance by increasing the editing of the document. The benchmark below is an example running about 14,000 edits. If you use the code below, you can easily increase the amount of change.

https://github.com/yorkie-team/yorkie/blob/main/test/bench/text_editing_bench_test.go

If a snapshot already exists, the document is rebuilt from the snapshot, so it would be good to test by increasing the snapshot creation cycle.

https://github.com/yorkie-team/yorkie/blob/8e15126d3491dd514125cd699631217d26e832a1/internal/cli/agent.go#L292-L297

2. How to write test code to make it arrive at the code I changed. I wrote test code referring to other one in TestSnapshot, but I noticed my code does not arrive at 195 line of pushpull.go. I don't know how to set up the situation in the test code.

When I ran the test I wrote, it reached that line. How about printing a log on the line?

[ejolie]

@ejolie Thanks for your contribution. Overall the code looks good.

1. How to determine the right chunk size of changes. Is there any way to test the performance depending on the chunk size of changes in my local environment?

I think we can test the performance by increasing the editing of the document. The benchmark below is an example running about 14,000 edits. If you use the code below, you can easily increase the amount of change.

https://github.com/yorkie-team/yorkie/blob/main/test/bench/text_editing_bench_test.go

If a snapshot already exists, the document is rebuilt from the snapshot, so it would be good to test by increasing the snapshot creation cycle.

https://github.com/yorkie-team/yorkie/blob/8e15126d3491dd514125cd699631217d26e832a1/internal/cli/agent.go#L292-L297

2. How to write test code to make it arrive at the code I changed. I wrote test code referring to other one in TestSnapshot, but I noticed my code does not arrive at 195 line of pushpull.go. I don't know how to set up the situation in the test code.

When I ran the test I wrote, it reached that line. How about printing a log on the line?

Thanks for your reply. I’d like to write benchmark test for “snapshot load changes test" to check the performance. So I tried to create activated clients to be used in the test by referring to createActivatedClients function in main_test.go but an error occurred when activating a client.

rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp [::1]:21201: connect: connection refused

I wonder if it is possible to use the client(client.Client) in the benchmark test code.

[hackerwins]

Unfortunately, I think it would be better to close this PR for now and open it when working again.