dgraph icon indicating copy to clipboard operation
dgraph copied to clipboard

fix(snapshot): Enable snapshot streaming after bulkload

Open all-seeing-code opened this issue 1 year ago • 1 comments

Description: Enable snapshot streaming after bulkload

Summary: Previously, the bulkloaded p directory couldn't stream to a new alpha due to a commitTs of zero. In this PR, the commitTs is sourced from the p directory, allowing the alpha to create a snapshot and subsequently stream it to another alpha.

Tests:

  • TestBulkLoaderSnapshotPDirinAlpha0: Load a p directory using bulkload. Then, initiate one alpha using the bulkloaded p directory and start a second alpha without any p directory. Query both alphas to ensure the snapshot has been successfully generated and that the data is accessible from both instances.
  • TestBulkLoaderSnapshotPDirinAll: Load a p directory using bulkload. Then, copy bulkloaded p directory in all alphas and start the cluster. Query all alphas to ensure that the data is accessible from all instances.
  • TestBulkLoaderDataLoss: Move the zero timestamp and load a p directory using bulkload. Then use this p directory on a fresh cluster (both zero and alpha are new). Validate that the query doesn't work without moving the timestamp of the new zero.

Closes: https://dgraph.atlassian.net/browse/DGRAPHCORE-214

Docs: NA

all-seeing-code avatar Aug 08 '23 09:08 all-seeing-code

We need more tests as we discussed, a few questions and minor comments are dgraphtest package

I have added more tests and extended the dgraphtest package to support some of the functions required for the new tests. This PR is ready for another round of reviews.

all-seeing-code avatar Aug 15 '23 11:08 all-seeing-code

This PR has been stale for 60 days and will be closed automatically in 7 days. Comment to keep it open.

github-actions[bot] avatar Jul 16 '24 14:07 github-actions[bot]