fiware-draco icon indicating copy to clipboard operation
fiware-draco copied to clipboard

feat: extend NGSI-LD support for NGSIToPostgreSQL processor (and other global upgrades)

Open bobeal opened this issue 3 years ago • 1 comments

Note: submitting the PR in draft status since there are still some things to finish (see TODOs below), but new features are already in good shape and tested on a real deployment.

Proposed changes

Globally extend the NGSI-LD support of the NGSIToPostgreSQL processor and also do some global upgrades (NiFi version, Java version):

  • chore: upgrade NiFi version to 1.14.0 (only tested for NGSIToPostgreSQL processor)
  • chore: set base Java version to 11 for nifi-ngsi-bundle
  • chore: replace part of System.out statements by log statements
  • feat: extract keys to ignore in payloads into a constant list for improved maintainability
  • feat: add support for multi-attributes (section 4.5.5 of the NGSI-LD spec) to avoid too long (and truncated) names
    • a configurable prefix can be removed from the dataset ids
  • feat: add support for sub-attributes of all types of attributes (was previously only supported for properties)
  • feat: use types as much as possible for columns (e.g. TIMESTAMPTZ for dates or NUMERIC for numeric values)
    • currently using TEXT (default), TIMESTAMPTZ and NUMERIC (more could be added in the future)
    • it won't break existing installations because columns already exist, but they could benefit from such a new feature

TODOs before removing the draft status

  • align NGSIv2 code where needed (e.g., in getValuesForInsert, listOfFields)
  • add some specific tests for the new features
  • upgrade bundle version (to which version?)

Types of changes

What types of changes does your code introduce to the project: Put an x in the boxes that apply

  • [ ] Bugfix (non-breaking change which fixes an issue)
  • [x] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

  • [x] I have read the CONTRIBUTING doc
  • [x] I have signed the CLA
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • [ ] I have added necessary documentation (if appropriate)
  • [ ] Any dependent changes have been merged and published in downstream modules

bobeal avatar Nov 02 '21 17:11 bobeal

I have read the CLA Document and I hereby sign the CLA

bobeal avatar Nov 02 '21 17:11 bobeal