azure-documentdb-changefeedprocessor-dotnet
azure-documentdb-changefeedprocessor-dotnet copied to clipboard
Azure
How to force a complete restart of a projection
I'm looking to restart a set of processing from the dawn of time [after processing for some time with a given prefix], discarding all progress (as opposed to the case discussed in #84 where one is indicatign the starting position one would like for a new lease collection (StartFromBeginning).
From my research, it seems:
- If I mint a new prefix, I seem to see all the documents (as one'd expect)
- if I use an existing lease, I can't seem to find any way via the API surface area to tell the processor I want to really Re start From The Beginning
- There is a deprecated
DiscardExistingLeasesthing which sounds like what I want to achieve
Workarounds that spring to mind for me:
- maintain
generationexternally, and bump that when I want to trigger reprocessing (and include that in theLeasePrefix) - perform surgery on the lease collection manually and/or with hack code as the API does not offer a way to achieve this
Am I missing something? If the only way I'm going to be able to achieve this effect is by doing one of the two things that spring to mind for me, then the xmldocs need updating to indicate that this feature does not do what it sounds like (either that, or if you can explain what use case this does fulfil, that'd work for me too)
@kadukf @mkolt Do we know why the original DiscardExistingLeases was deprecated?
It’s a dangerous scenario because it’s very easy to kill all leases by accident and thus lose all progress on the feed. We can consider bring it back, but yes, it’s dangerous. Thanks, Michael
From: Matias Quaranta [email protected] Sent: Friday, January 18, 2019 8:28 AM To: Azure/azure-documentdb-changefeedprocessor-dotnet [email protected] Cc: Michael Koltachev [email protected]; Mention [email protected] Subject: Re: [Azure/azure-documentdb-changefeedprocessor-dotnet] How to force a complete restart of a projection (#123)
@kadukfhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkadukf&data=02%7C01%7Cmkolt%40microsoft.com%7Cc6ba594abab04621b14608d67d61f3ec%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636834256989338597&sdata=lgMKxdRx6j76tXIz3XeHC1pUx%2F21FfP%2F5ncfzP0yruc%3D&reserved=0 @mkolthttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmkolt&data=02%7C01%7Cmkolt%40microsoft.com%7Cc6ba594abab04621b14608d67d61f3ec%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636834256989348579&sdata=KJ90LYJx8Pt6PNEKWaobDceTFC717r0InC7szkJtgWs%3D&reserved=0 Do we know why the original DiscardExistingLeases was deprecated?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-documentdb-changefeedprocessor-dotnet%2Fissues%2F123%23issuecomment-455605501&data=02%7C01%7Cmkolt%40microsoft.com%7Cc6ba594abab04621b14608d67d61f3ec%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636834256989358588&sdata=EQ4ZoIjcXYXboU0NktmbuuXcs1kME5m0%2B4jARbF6ySo%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAQnQBGH2bht1ghQUFby7dhAuEaUa2clbks5vEfYggaJpZM4aDdBJ&data=02%7C01%7Cmkolt%40microsoft.com%7Cc6ba594abab04621b14608d67d61f3ec%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636834256989358588&sdata=%2FedHVy9zutP%2BGuU9wZTZPWLU3DlB5PH58i869eVzc%2Fo%3D&reserved=0.
I'm not suggesting it should be equally prominent in the builder DSL or a general options record - clearly a destructive thing like this should not share space in the docs with stuff that's actually talking about a semantic defining some consistency one is expecting.
I think my ideal surface-level API in this space would be be to have a top level Method with something like Destroy or Dangerous in the name.
It's definitely important to have to have such a function though, whether that be in the form of
- a "
Task DiscardAllProgress()on the processor as a first class thing - a separated one-off call which intentionally lives somewhere else given how you might be pulling the rug out from sister-projectors in a cluster.
Having that also allows xmldocs like the one that prompted #84 to point somewhere as a counterpoint to underscore the purpose of something.
@bartelink Why not just delete your lease and set the StartFromBeginning = true? As long as you've named your leases using the LeasePrefix option, it is straight-forward to find the two docs that must be deleted to wipe out a lease.
@jiffypopjr Yes, I could - the reason I'm asking here though is:
a) there was once such a facility b) others demonstrably need it c) not having it makes the API confusing and incomplete
Hence I'm seeking some insight as to whether there are plans to reinstate and/or whether e.g. a PR to add such a facility in any manner would be admitted.
@ealsur @kadukf @mkolt right now, I'm looking at having to build something to delete the state which will need to stay in sync with the CFP impl.
If I am to do this, I'd much prefer to do that as a PR here - I'd really appreciate some information as to whether such a PR would be accepted and/or whether there is a plan to provide this at any point.
While there is room to argue that if one wanted to reproject from the start, that one should version/burn the LeaseId for this projection it would not reflect the intention - I want to
a) discard the progress accumulated by this lease-set and
b) put it beyond use
c) have a guarantee that each processor will restart
The other point I'll make again, is that, while aspect c is hard to deliver, not having this facility makes it harder to understand the API for someone thinking of this as being like any other projector/consumer progress thing out there; thus if the answer is "no, this package will not provide such a facility for the foreseeable future", I'd be at least insisting on the readme.md mentioning this as being out of scope.
I'm intending to keep this issue open until there's a clear response on this - I need to provide a clear answer as to how to accomplish the standard "blow away the projected state and replay" requirement given I'm using an abstraction that attempts to encapsulate such concerns in full.
Hi Ruben, how about slighly different approach: provide a helper method to delete existing leases for given lease collection info + ( lease prefix or CFP options) ?
From: Ruben Bartelink [email protected] Sent: Wednesday, February 6, 2019 6:50 PM To: Azure/azure-documentdb-changefeedprocessor-dotnet [email protected] Cc: Michael Koltachev [email protected]; Mention [email protected] Subject: Re: [Azure/azure-documentdb-changefeedprocessor-dotnet] How to force a complete restart of a projection (#123)
@ealsurhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fealsur&data=02%7C01%7Cmkolt%40microsoft.com%7Cfbde0c7435a14c3e391c08d68ca6f6d7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636851046077301823&sdata=IBm2W3xl182%2FljlfyPzgCJjKDuYeXvvEnNO7L8pXlZE%3D&reserved=0 @kadukfhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkadukf&data=02%7C01%7Cmkolt%40microsoft.com%7Cfbde0c7435a14c3e391c08d68ca6f6d7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636851046077301823&sdata=x0Fn8TYZG7o29%2Fp4%2F66ISA83UO7O4FXQ4R1J6ssrqPI%3D&reserved=0 @mkolthttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmkolt&data=02%7C01%7Cmkolt%40microsoft.com%7Cfbde0c7435a14c3e391c08d68ca6f6d7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636851046077311823&sdata=FQz%2BiPrIPdIaS%2FBDJ2weo%2FJbMFj45hYzjMw%2FqenNeY0%3D&reserved=0 right now, I'm looking at having to build something to delete the state which will need to stay in sync with the CFP impl.
If I am to do this, I'd much prefer to do that as a PR here - I'd really appreciate some information as to whether such a PR would be accepted and/or whether there is a plan to provide this at any point.
While there is room to argue that if one wanted to reproject from the start, that one should version/burn the LeaseId for this projection it would not reflect the intention - I want to a) discard the progress accumulated by this lease-set and b) put it beyond use c) have a guarantee that each processor will restart
The other point I'll make again, is that, while aspect c is hard to deliver, not having this facility makes it harder to understand the API for someone thinking of this as being like any other projector/consumer progress thing out there; thus if the answer is "no, this package will not provide such a facility for the foreseeable future", I'd be at least insisting on the readme.md mentioning this as being out of scope.
I'm intending to keep this issue open until there's a clear response on this - I need to provide a clear answer as to how to accomplish the standard "blow away the projected state and replay" requirement given I'm using an abstraction that attempts to encapsulate such concerns in full.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-documentdb-changefeedprocessor-dotnet%2Fissues%2F123%23issuecomment-461270199&data=02%7C01%7Cmkolt%40microsoft.com%7Cfbde0c7435a14c3e391c08d68ca6f6d7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636851046077321836&sdata=5YlSFk9Uvk1h8AZzmCfYtWKZVRACr2IbW9pvBe8AIsE%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAQnQBChG9FuMnpihgbcIQhqntEp9AN6sks5vK5RdgaJpZM4aDdBJ&data=02%7C01%7Cmkolt%40microsoft.com%7Cfbde0c7435a14c3e391c08d68ca6f6d7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636851046077321836&sdata=A2BlQylyVIXfujJMmJ1JOC0WQ51jgJ7UJZTVHIt%2FXGk%3D&reserved=0.