snowplow
RDB Loader: discover ssh tunnel local port automatically
Whether it is single or multiple loader running together on the same host, only requirement on ssh tunnel's local port is not being allocated on the host the app is running on.
Loader could pick an unallocated port so that user doesn't even need to provide a port. This could also prevent cases where user provides an allocated/invalid port number.
If we need it configurable for a reason, we can at least improve config decoding to make sure app starts with a valid & unallocated port.
https://github.com/snowplow/snowplow-rdb-loader/blob/c7126f4e7f32795d7949e048192b7a5b032980d9/modules/common/src/main/scala/com/snowplowanalytics/snowplow/rdbloader/common/config/StorageTarget.scala#L175
So what you're saying is that we can make it Option[Int] and if it's not provided try to pick a random one?
My first idea is to remove the config field localPort so that user doesn't even think about it. In this case, loader picks a random valid one.
If we need it configurable for a reason, then we should improve decoding to make sure provided Int is an unallocated and a valid port number.
Option[Int] is another good idea but I didn't think of it initially
user doesn't even think about it. In this case, loader picks a random valid one.
What if the port is closed via security group? The point of the explicit number is that user knows what is open and what's closed.
then we should improve decoding to make sure provided Int is an unallocated and a valid port number.
Decoding is one step, completly pure - string to an object, allocation is another step, side-effectful. One must never mix two.
What if the port is closed via security group? The point of the explicit number is that user knows what is open and what's closed.
This is a local port on the local host - it is not a port used outside of the host. When you create a tunnel you are assigning a local random port which forwards to a known port on the bastion host.
This should never be an issue!
If checking whether port is allocated or not should not be part of decoder, at least checking the validity could be. I don't think loader should accept an invalid port number which would inject invalid state to the program.