nifi
nifi copied to clipboard
apache
NIFI-10235 Set Replay ContentClaim Length from Content Repository
Summary
NIFI-10235 Changes the behavior of Provenance Replay handling to resolve issues when running with encrypted repositories enabled.
When running with encrypted repositories, the content stored in the repository is larger than the file size itself, since it contains an encryption metadata header. This difference in size causes Provenance Replays to fail due to setting the replay ContentClaim length based on the previous file size. Changing the approach to set the ContentClaim length based on the ContentClaim size that the Content Repository reports allows replays to function as expected.
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking
- [X] Apache NiFi Jira issue created
Pull Request Tracking
- [X] Pull Request title starts with Apache NiFi Jira issue number, such as
NIFI-00000 - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such
NIFI-00000
Pull Request Formatting
- [X] Pull Request based on current revision of the
mainbranch - [X] Pull Request refers to a feature branch with one commit containing changes
Verification
Please indicate the verification steps performed prior to pull request creation.
Build
- [X] Build completed using
mvn clean install -P contrib-check- [X] JDK 8
- [ ] JDK 11
- [ ] JDK 17
Licensing
- [ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
- [ ] New dependencies are documented in applicable
LICENSEandNOTICEfiles
Documentation
- [ ] Documentation formatting appears as expected in rendered files
@exceptionfactory Working to recreate setup before and after fix...
When trying instructions for creating storetype of type BCFKS https://exceptionfactory.com/posts/2021/11/10/configuring-apache-nifi-repository-encryption (under SECRET KEY GENERATION) the keytool -genseckey followed by the keytool -list command has output saying that there is an invalid MAC
This is not a blocker on this (as ticket was done with PKCS12), but I noticed you were the author of that article so maybe that could be looked into at some later time (and by the way, thank you for your very informative articles in general :) )
When trying instructions for creating storetype of type BCFKS https://exceptionfactory.com/posts/2021/11/10/configuring-apache-nifi-repository-encryption (under SECRET KEY GENERATION) the keytool -genseckey followed by the keytool -list command has output saying that there is an invalid MAC
Thanks for the feedback @NissimShiman!
When verifying the BCFKS keystore, it is necessary to provide the store password in the command. The example has the word PLACEHOLDER, which must be changed to the actual password.
I am still evaluating the issues associated with encrypted repositories. The current version of this pull request resolves the problem for simple use scenarios, but there is an additional issue related to content claim length determination when the framework writes multiple FlowFiles to a single resource. I am planning to update the pull request as soon as I have a working solution.
@exceptionfactory The initial fix works as I was able to verify (using a pkcs12 keystore) that trying to replay a SEND for InvokeHTTP (to ListenHTTP) did not work before the fix, but does work after this fix.
I see where the code is now looking at ContentClaim size to get a more accurate size. Very nice diagnosis.
LGTM +1, but I'll plan to circle back if/when additional work is done.
The BCFKS issue is even when using a real password (and doesn't occur when following the instructions for PKCS12 keys - found in the first link of the SECRET KEY GENERATION section), so I am not sure what it happening with that.
Error is: keytool error: java.io.IOException: BCKFS KeyStore corrupted: MAC calculation failed
using java: openjdk version 1.8.0_332
Thanks again for the feedback on this @NissimShiman, closing for now and will reopen pending a more comprehensive solution.