New Processor: Datetime to string

[bossenti]

Body

Description

A user should have the ability converta datetime values to a human-readable string value in a pipeline. Therefore, a new processor is required that provides the functionality to convert a datetime value to a string value based on a given string format.

Example Configuration

StringFormat : A format string that describes the format of the datetime string

The string format should be configurable with a conveniant UI dialog, ideally similar to our Notification sink:

Additional Functionality

If you have any other ideas for the functionality of this new processor, please feel free to add them here.

Mentoring

As this issue is marked as good first issue: one of @dominikriemer, @tenthe, or @bossenti are happy to provide help for getting started, just tag (one of) them if you want to start working on this issue and need some help. This issue is ideal for people that are interested in getting started with StreamPipes and its SDK.

StreamPipes Committer

I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.

[lab596]

Hey @bossenti, I am honestly pretty new to coding but I would love to contribute to this project if I got a chance. Taking a look at this issue, some of the concepts may be over my head but I would like to give it a try. For now I have cloned the repo however when I run the code I seem to be getting this message as I believe I have not set up my environment:

Checking if consul is available on host consul and port 8500
Could not connect to Consul instance...

I have taken a look at SteamPipes CLI however I seem to be stuck on the streampipes in the CLI. Whenever I try to run a command I see bash: streampipes: command not found but I'm not sure what I'm missing.

[lab596]

Hey, I believe I found the hiccup, in the documentation it states to utilize the ./streampipes command in bash to access the streampipes CLI, however this forgets to take into account the file path, Instead the env was able to be initiated correctly using ./installer/cli/streampipes. Please let me know if this is accurate, I am able to update the documentation if needed.

[lab596]

Whenever I try to set up an environment I am faced with the following error:

$ streampipes env -s pipeline-element
[INFO] Set environment to 'pipeline-element'
cp: target 'Projects/streampipes/installer/cli/.spenv' is not a directory

@bossenti @dominikriemer @tenthe Any help is appreciated, thanks!

[bossenti]

Hi @lab596,

First of all, we really appreciate your willingness to contribute 🙂 Don't be afraid if you are not having that much experience, we are happy to assist you.

Please let me know if this is accurate, I am able to update the documentation if needed.

Yes, you are right. The CLI commands should be executed in the directory ./installer/cli/. If the documentation does not explain this properly, a PR is welcome. Please be aware that our documentation is maintained in a dedicated repository: https://github.com/apache/streampipes-website

[bossenti]

cp: target 'Projects/streampipes/installer/cli/.spenv' is not a directory

Do you observe this error when executing the CLI command in ./installer/cli?

[lab596]

Do you observe this error when executing the CLI command in ./installer/cli?

Yes, I have tried multiple methods including using the full command ./installer/cli/streampipes env --set pipeline-element as well as setting installer\cli as a windows path and running the command above: streampipes env -s pipeline-element. Both lead to the same error:

[INFO] Set environment to 'pipeline-element'
cp: target 'Projects/streampipes/installer/cli/.spenv' is not a directory

[bossenti]

I just gave it a shot on my Windows machine as well. On my end running ./installer/cli/streampipes env --set pipeline-element completes successfully with below output:

[INFO] Set environment to 'pipeline-element'

Looks like there is a hick-up with respect to the path where the config should be written to. Can you please execute echo $STREAMPIPES_WORKDIR and share the result? 🙂

[lab596]

Please be aware that our documentation is maintained in a dedicated repository: https://github.com/apache/streampipes-website

@bossenti, I have created a PR in that repo with a few minor changes please let me know if that correctly updates the documentation. Additionally, I believe that documentation also has to be updated on this repo here.

[lab596]

Can you please execute echo $STREAMPIPES_WORKDIR and share the result? 🙂

I simply get an empty line:

$ echo $STREAMPIPES_WORKDIR

[bossenti]

Okay, that's expected, We just set it while the script is executed I was just wondering if this variable is somehow overwritten.

Since I'm running out of ideas, maybe @dominikriemer @tenthe or @RobertIndie are able to support here

[lab596]

Okay, thanks for the update.

Actually, I'm not sure if I need this environment. Can someone please clarify? This is the original error I was faced with:

2023-09-09T10:24:16.866-07:00  INFO   --- [           main] o.a.s.s.consul.ConsulProvider            : Checking if consul is available on host consul and port 8500
2023-09-09T10:24:16.866-07:00  INFO   --- [           main] o.a.s.s.consul.ConsulProvider            : Could not connect to Consul instance...
2023-09-09T10:24:16.866-07:00  INFO   --- [           main] o.a.s.s.consul.ConsulProvider            : Retrying in 1 second

I just originally assumed that is because of the environment setup. Any additional info is appreciated.

[dominikriemer]

@lab596 This error typically occurs when you are running an extensions service locally, while Consul (and other services) run in a container. You can set two environment variables SP_DEBUG=true and SP_CONSUL_HOST=localhost to resolve this issue for local development.

[lab596]

Hey, I believe I tried this by running this in the terminal. However, nothing happened. If this is not what you meant can you please elaborate?

export SP_DEBUG=true
export SP_CONSUL_HOST=localhost

[bossenti]

@lab596 to better support you here, I'd like to ask what you are trying to achieve. Do you just want to start StreamPipes to try it out? Or, do you want to get the development setup of StreamPipes ready on your machine?

[lab596]

@bossenti I have successfully started and explored StreamPipes by following the basic installation instructions here.

However, the problems I am facing is when trying to deploy a development setup by following the instructions here and by extension the CLI setup as outlined here.

[dominikriemer]

@lab596 thanks for clarifying! The setup should work out of the box - here is a short description what I do when developing a pipeline element:

• Stop the docker-compose instance of the basic installation (you might also consider to clean this instance by executing docker-compose down -v.

• To set up the development environment, go to installer/cli, configure a template (./streampipes env --set pipeline-element) and run ./streampipes up -d.

• You should then have a running StreamPipes instance including the UI with backend ports which are configured for local development.

• When you open the IDE of your choice, create a new pipeline element and execute the Init method (e.g., try with the extensions-all-jvm module. Here is the place where you need to configure the environment variables as startup env variables (SP_DEBUG and SP_CONSUL_HOST). This will internally configure the extensions service for local development.

• If you have any problems, you can check the logs of the extensions service (there should be a log message which shows where the system tries to connect to the core to register pipeline elements).

Hope this helps! Please ask if anything is unclear.

[lab596]

Hey @dominikriemer,

Thanks for all the help. I was able to set up most of the development setup using your instructions however when attempting to create a new pipeline element to test it out I was facing this error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/streampipes/connect/iiot/ConnectAdapterIiotInit
        at org.apache.streampipes.extensions.all.jvm.AllExtensionsInit.provideServiceDefinition(AllExtensionsInit.java:48)       
        at org.apache.streampipes.service.extensions.StreamPipesExtensionsServiceBase.init(StreamPipesExtensionsServiceBase.java:51)
        at org.apache.streampipes.extensions.all.jvm.AllExtensionsInit.main(AllExtensionsInit.java:40)
Caused by: java.lang.ClassNotFoundException: org.apache.streampipes.connect.iiot.ConnectAdapterIiotInit
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
        ... 3 more

What I first did was ensure to add the startup env variables by running (I'm trying to run extensions-all-jvm as suggested):

$ docker run -e SP_DEBUG=true -e SP_CONSUL_HOST=localhost -d apachestreampipes/extensions-all-jvm:0.93.0-SNAPSHOT
701e4b3b072da950b980bf2bd893365912bb99fe4e51bc963d97a1323a08ee12

Then I attempted to run the AllExtensionsInit.java within the streampipes-extensions-all-jvm folder.

Am I missing something?

[bossenti]

@dominikriemer