Add API versioning to migrations between environments

[DeamonDev]

💥 Proposal

Currently the API for migrations between environments is not versioned. Since the process of migration involves two deployed Nu instances taking to each other, it may cause problems with communication between them when API changes in next version. I propose to version the API to one version back,

[mk-software-pl]

1. What does the migration do?
2. What do we migrate?
3. How does the process of migration look like?

[DeamonDev]

Are you sure all of these questions are relevant within the scope of this issue? I mean, here we are dealing with boundary of some bounded context where I think we should at least treat actual migration process of the target backend as black box. Nevertheless I will try to answer your questions.

The process of migration of scenario between two environnents involves frontend of source Nussknacker, backend of source Nussknacker and backend of target Nussknacker. This should shed some light on your question about what the migration do. It just migrate the scenario from primaryEnvironment to secondaryEnvironment. I will refer to them as source / target environments. Currently the source backend more or less just forwards the requested data from frontend (containing for example scenario graph) and the target backend is the one who executes the actual migration. The problem is when two backends runs on different versions and the requested data might turn out to be incompatible. Typical issue is newer version of Nussknacker requires more fields as its request data and older (source) Nussknacker cannot be aware of this fact.

There are two natural solutions how to deal with that. We may set up external proxy which will do the actual translation. Or we can version the API manually by storing multiple DTOs. Each of them has its pros and cons: the proxy approach does not require additional DTOs in code but required external module which has to be maintained. The end to end testing also may become more problematic. The pros and cons of manual versioning are basically reverse of the pros/cons of the proxy solution.

Each of these approaches may be scalled in various directions. The proxy module may transform DTOs from other subdomains and their corresponding APIs while manual versioning may track DTOs back in time up to one, or more versions.

I hope I described it without unnecessary technical complexities. Let me know what you think :)

[mk-software-pl]

Wow, this is super relevant. If I have to review the code I need to know what the feature does and what are the requirements. TBH any feature should start with these.

Additional questions:

1. do we want to migrate scenarios from the older version to the newer version only, right?
2. let's suppose the current version is 1.14. In this version, we should maintain the current migration request DTO, and the previous version migration request DTO, right? It means that in the next version (1.15) the current migration request from v1.14 becomes the previous migration request DTO and the current one should be defined. Am I right?

[DeamonDev]

Yeah, I just thought you want to know the internals of migration. So we are on the same page.

1. I assumed bidirectional flow. Altough in the description above I assumed scenario where source has older version but it does not have to be the case.
2. Yes, you are right. It may be summarized as:

1.14	1.15	1.16
A	B	C
B	C	D

[coutoPL]

This approach seems to be a little bit problematic.

Let's consider another approach: the migration request will have:

1. version of the scenario (V1)
2. the scenario in the format specified by the version (V1)

Then the service that is responsible for converting the scenario from V1 format to V2 (the format of a given instance). So we have to provide a conversion: V1 -> V2. We could easily maintain conversion between several versions.

Eg. instance 1: v1.12, instance 2: v1.14

migration request from instance I1:

POST /migrate 
{ 
  "version": "1.12",
  "scenario":  {
    // format specific for the v1.12
  }
}

in instance I2 the request is being handled. The instance has the following conversions:

1. v1.12 -> v1.13
2. v.1.13 -> v1.14

To migrate v1.12 to v1.14 both conversions have to be applied.

When we develop the v1.15 the new conversion will be added (no modifications, just a new thing is added). Some test will keep an eye on the existence of proper conversion (to make sure the new conversion is added for a new version).

WDYT?

[DeamonDev]

Ok, if we want to version more API versions then it is better solution. My approach was just a hack (possible because I versioned only one version back in the time). So I propose to define

sealed trait NuVersion { 
  def prevVersion: Option[NuVersion]
  def nextVersion: Option[NuVersion]
  }

case object Version1_14 extends NuVersion
case object Version1_15 extends NuVersion

.... And use this to be able to perform transitive DTO translations.