nifi icon indicating copy to clipboard operation
nifi copied to clipboard

Published 20 hours ago • verified publisher icon apache

NIFI-8248 Modify PutAzureDataLakeStorage processor to use temp file i…

Open timeabarna opened this issue 3 years ago • 6 comments

Summary

NIFI-8248 enables the usage of a temporary file instead of immediate overwrite.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Pull Request Tracking

  • [x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
  • [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

  • [x] Pull Request based on current revision of the main branch
  • [x] Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

  • [x] Build completed using mvn clean install -P contrib-check
    • [x] JDK 8
    • [ ] JDK 11
    • [ ] JDK 17

Licensing

  • [ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
  • [ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

  • [x] Documentation formatting appears as expected in rendered files

timeabarna avatar Jun 27 '22 05:06 timeabarna

Hello @jfrazee and @exceptionfactory,

I've managed to get to modifying PutAzureDataLake processor upload. Last year we had a long discussion how different services like Spark or HDFS could be impacted by this change. Based on that discussion, can be found here, the following design decisions have been made for this PR:

List processor: - A boolean property has been created to turn on or off showing temp directory in list results. If the list processor would hide files starting with '.' or '_' set to default true, users listing hidden files now would be impacted. If the default value would be false, temporary files would appear in listing which may again be an impact for users.

Put processor: - A property for temp directory location has been created. - Temp files created under a static folder with a generated UUID and the existing file name.

When you have some time can you please check? Thanks in advance for your help.

timeabarna avatar Jun 27 '22 06:06 timeabarna

Hello @jfrazee and @exceptionfactory,

When you have some time can you please check this PR? Thanks in advance for your help.

timeabarna avatar Jul 11 '22 07:07 timeabarna

Hello @jfrazee and @exceptionfactory,

When you have some time can you please check this PR? Thanks in advance for your help.

timeabarna avatar Aug 09 '22 05:08 timeabarna

Thanks for the reminder @timeabarna, will plan on reviewing this soon.

exceptionfactory avatar Aug 09 '22 12:08 exceptionfactory

Hello @exceptionfactory,

Thank you very much for your review. I've modified the code based on your recommendations.

timeabarna avatar Aug 11 '22 07:08 timeabarna

Thanks @exceptionfactory, I've updated the PR.

timeabarna avatar Aug 12 '22 06:08 timeabarna

Good catch @jfrazee, I totally overlooked this. I've updated the PR with the corrected documentation. Thank you very much for your help.

timeabarna avatar Aug 19 '22 06:08 timeabarna