Create SlicingDice sink

[joaosimbiose]

This PR resolves the issue #1541

The SlicingDice's development team wrote this Cygnus extension for a new sink to persist data in SlicingDice's databases.

Please, we kindly ask you to review this pull request and provide us your insights and considerations about the implementation.

We read and followed all the contribution guidelines [1], documenting the code and also creating all the necessa1y tests.

[1] - https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/contributing/contributing_guidelines.md#contributing-guidelines

Thanks in advance.

---

SlicingDice API Overview

SlicingDice provides a REST API [2] that allows its customers to insert (and query) data using the API.

In order to insert values into a SlicingDice database, you send a POST request to https://api.slicingdice.com/v1/insert with a JSON object which the key is the entity ID containing JSON objects with database's column names and their respective values.

In the Cygnus case, our implementation considers the entity ID to be the IoT device generating the data.

Any timeseries-related data (metrics) sent by Cygnus will be stored in a SlicingDice database on an Event Column and the non-timeseries data on an Attribute Column.

An example of the API insertion request generated by this Cygnus extension would look like this:

curl -X POST https://api.slicingdice.com/v1/insert \
    -H 'Authorization: DATABASE_API_KEY' \
    -H 'Content-Type: application/json' \
    -d '{
        "DeviceID": {
            "dimension": "default",
            "sensor-type": "Types of the Sensor",
            "sensor-model": "Model XYZ",
            "sensor-firmware-version": "1.3.2",
            "temperature": [
                {
                    "value": 78,
                    "date": "2018-11-14T12:54:12Z"
                }
            ],
            "pression": [
                {
                    "value": 13,
                    "date": "2018-11-14T12:54:12Z"
                }
            ],
            "moisture": [
                {
                    "value": 20,
                    "date": "2018-11-14T12:54:12Z"
                }
            ],
            "sensor-status": [
                {
                    "value": "active",
                    "date": "2018-11-14T12:54:12Z"
                }
            ]
        },
        "auto-create": ["dimension", "column"]
    }'

Implementation considerations:

• SlicingDice's API supports HTTP and HTTPS.
• SlicingDice's Cygnus extension implementation is by default leveraging the SlicingDice's ability to create new columns and dimension as needed, if they don't exist on the database, by adding the parameter "auto-create": ["dimension", "column"] at the end of all insertion requests. This allows this integration to never break in case Cygnus adds new columns.

[2] - SlicingDice's API Docs are available at https://docs.slicingdice.com

Implementation Overview

feature

[cygnus-common] com.telefonica.iot.cygnus.backends.slicingdice.SlicingDiceBackendImpl

• This class extends HttpBackend.
• createColumns()
  • This method will create all the necessary SlicingDice columns if the user doesn't specify the auto_create=true configuration.
• insertContextData()
  • This method stores the data on SlicingDice servers using the REST API.

[cygnus-ngsi] com.telefonica.iot.cygnus.sinks.NGSISlicingDiceSink

• This class extends NGSISink.
• The configurable variables are below:

variable	default	description
database_key		the database key used to connect to SlicingDice
auto_create	true	if true we will create the columns automatically on SlicingDice as we insert

Dockerfile, agent.conf and cygnus-entrypoint.sh

• Dockerfile, agent.conf and cygnus-entrypoint.sh are edited to be able to use NGSISlicingDiceSink.

[fgalan]

Thank you very much for the contribution!

I have had a look to the PR. With regards to code I haven't performed an in deep review (I'm afraid I lack the knowledge in SliceDice to do that ;). I have only look for kind of "formal" aspect, with some minor comments regarding it. As long as the new sink doesn't break the existing e2e regression (we would check upon merging), that's ok.

My main piece of feedback is regarding documentation. The work you have done needs to be properly and comprehensively documented in the PR along with the code and unit test contribution. This includes:

• Main README.md, to include SliceDice in the list of supported sinks (i.e. a new bullet in "NGSI-like context data in:" list)
• Add a new file slicedice_backend.md to doc/cygnus-common/backends_catalogue/ directory, properly linked from the README.md in that same directory. Please, have a look to the other files in the same directory and use the same content structure.
• Add a new file slicedice_sink.md to doc/cygnus-ngsi/flume_extensions_catalogue/ directory, properly linked from the README.md in that same directory ("Sinks" section). Please, have a look to the other xxx_sink.md files in the same directory and use the same content structure. This is the place where the table with cygnus-ngsi.sinks.slicingdice* parameters is provided, in addition to examples.
• Modify doc/cygnus-ngsi/installation_and_administration_guide/install_with_docker.md to include the env vars used by Dockerfile/cygnus-entrypoint.sh and their description. Withing "Using a specific configuration" add a new bullet for your sink.
• Add the new files to mkdocs.yml so they can be also rendered in the RTD version of the documentation (https://fiware-cygnus.readthedocs.io/en/master/)

This list could not be exahustive. It would be great if you could have a look to the Cygnus .md files in general in the repository to find any other possible directory/file that could need additions and I have missed, please.

Thanks!

[gmenegatti]

Hi @fgalan,

Thanks so much for the review and all the comments.

We will correct everything ASAP and let you know.

Regards, Gabriel.

[fgalan]

@joaosimbiose any update in this PR, please? Don't hesitate of asking if something is not clear about what is pending in this PR to be finished and merged. Thanks!