Implement Action Versioning

[jiangpengcheng]

Description

Add action versioning feature for actions

Design

Design document: https://cwiki.apache.org/confluence/display/OPENWHISK/Action+Versioning

Newly created/updated actions will include a version in their document id, like whisk.system/[email protected], so they will not replace old versions.

And to be compatible with current actions whose id doesn't contain version info, I added a new view action-versions in CouchDB which can use action's fullyQualifiedName to get all its versions and related ids, and it will return the default version of the action when includeDocs=true, so with one query request, system can get the default version(optional) and version-ids mappings for an action:

function (doc) {
  var isAction = function (doc) { return (doc.exec !== undefined) };
  if (isAction(doc)) try {
    var value = {
      _id: doc.namespace + "/" + doc.name + "/default",
      namespace: doc.namespace,
      name: doc.name,
      version: doc.version,
      docId: doc._id
    };
    emit([doc.namespace + "/" + doc.name], value);
  } catch (e) {}
}

~There is a new annotation field publish for action, action owners can set it to false for test-only purpose to make sure users can not get/invoke non-published versions, and set it to true when action is well developed~

Before update/get/invoke/delete an action, this view will be queried first to get the docId of action, and then using the docId to fetch action from database:

• create/update:

1. if defaultVersion is passed and is a valid SemVer, system won't create a new action but just create/update the default version entry for this action with $fullyqualifiedName(false)/default as docId. if defaultVersion is not passed, goto step2
2. system will query action-versions view, and if version count exceed max allowed value, reject request. if not exceed, goto step3
3. get a suitable docid to update, if default version is set before, return default version's id, else, return latest version's id, goto step4; if no version exist at all, return None, goto step5.
4. create a new action based on old action got in step3 and user's input, with version = version.upPatch
5. create a new action based on user's input, with version=0.0.1

• get:

1. if showVersions is true, return versions list of the action, else, goto step2:

   case class WhiskActionVersionList(namespace: EntityPath,
                                     name: EntityName,
                                     versionMappings: Map[SemVer, DocId],
                                     defaultVersion: Option[SemVer])
   

2. get docId for the given action'name and version parameter, if version is not provided, the default or latest version(when default version is not set) will be choosed, then fetch action using this docId

• invoke:

1. get docId for the given action'name and version parameter, if no version is specified, the default or latest version will be choosed
2. pass the docId to ActivationMessage
3. invoker can get action using docId passed with ActivationMessage directly, so it doesn't need to care about how to transform version to docId

• delete:

1. if version is provided, delete related version, else, goto step2
2. get version list of action using design doc action-versions, goto step3
3. if there are multiple versions exist and deleteAll=false, reject request, else goto step4
4. delete all versions, and return the latest version of action

With this view, the old style of docId for actions is also compatible with this PR

And for performance reason, there is a cache layer for querying this view from database

Other than actions, there is also some tiny changes to trigger and rule, since in current master, trigger, rule and action can not using same name by follow same docId style, while with this PR, actions will use a different way for their docId(name + version), so triggers and rules may use same name with some actions.

So before save trigger and rule, it will also fetch the action-versions view first, and if there are some docIds returned, which means that entity name is occupied, openwhisk should return a Conflict error with message Resource by this name exists but is not in this collection.

TODO

• [ ] support action versioning for webactions

• [ ] support action versioning for sequence actions and conductor actions

• [ ] support action versioning for trigger and rule

Related issue and scope

• [x] I opened an issue to propose and discuss this change (#4580)

My changes affect the following components

• [ ] API
• [x] Controller
• [ ] Message Bus (e.g., Kafka)
• [ ] Loadbalancer
• [ ] Invoker
• [ ] Intrinsic actions (e.g., sequences, conductors)
• [x] Data stores (e.g., CouchDB)
• [ ] Tests
• [ ] Deployment
• [x] CLI
• [ ] General tooling
• [ ] Documentation

Types of changes

• [ ] Bug fix (generally a non-breaking change which closes an issue).
• [ ] Enhancement or new feature (adds new functionality).
• [x] Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

• [x] I signed an Apache CLA.
• [x] I reviewed the style guides and followed the recommendations (Travis CI will check :).
• [x] I added tests to cover my changes.
• [x] My changes require further changes to the documentation.
• [ ] I updated the documentation where necessary.

[bdoyle0182]

so many exciting PR's at once :)

[bdoyle0182]

Can we add creating a max version limit to todo? This is especially important if not supplying a version creates a new version on every action post, users will end up creating a lot of versions. Should be easy to have the check if we already have the action version view.

[jiangpengcheng]

Can we add creating a max version limit to todo? This is especially important if not supplying a version creates a new version on every action post, users will end up creating a lot of versions. Should be easy to have the check if we already have the action version view.

I will implement it in this PR

[bdoyle0182]

"if there is no docId, just created a new action based on user's input and save it"

Does this imply that if an entirely new doc id / action is created without a version supplied, that it will be created without a version? Or is it created as 0.0.1 so that all new actions post update will be created with a version?

[bdoyle0182]

LGTM after my comments. I don't know what the approval process for such a substantial feature like this should be, but I will approve once one or two more people have gone through this. I'm impressed you were able to implement a feature like this and maintain backwards compatibility without the code getting out of control. This is a feature we've wanted for years.

One thing to add about backwards compatibility, could we create a migration script that adds version @0.0.1 to the db and version view for every non-versioned action. Once upgraded, we would like to do something like that so that we don't have a mixture of the two types of data to deal with. And if such a script is provided, we can remove the backwards compatibility support in a subsequent release to keep the codebase clean and direct operators to use the script once upgraded past this commit.

[jiangpengcheng]

"if there is no docId, just created a new action based on user's input and save it"

Does this imply that if an entirely new doc id / action is created without a version supplied, that it will be created without a version? Or is it created as 0.0.1 so that all new actions post update will be created with a version?

a 0.0.1 will be used

[jiangpengcheng]

code looks kind of ugly to make it compatible with old style, I will write a script to copy current docs to new docs

[jiangpengcheng]

I wrote a wiki in https://cwiki.apache.org/confluence/display/OPENWHISK/Action+Versioning to help understanding this PR

[ningyougang]

• will provide version list query feature with cli tool in future? e.g. wsk actionVersion $action list or wsk action version $action list
• it is better to update the document
  • Change the create action flow's withDefaultVerstion -to withVersionParameter
  • Add the publish logic to the document

[upgle]

@jiangpengcheng please resolve conflicts

[Deklin]

Just to add some comments to this. It would be beneficial to be able to run an action with a @version tag as specified

e.g. /[email protected]

However it is also important that /helloworld and perhaps helloworld/@latest always run the latest published version. I believe other FAAS services like fission do versioning this way and it makes contractual binding from consumers much easier.

I can either point to a specific version if I am in a strictly controlled environment, or can always point to @latest to get the latest published version of the code.

--edit

It does look like the design document handles this scenario in the invoker :) Thank you

[jiangpengcheng]

• will provide version list query feature with cli tool in future? e.g. wsk actionVersion $action list or wsk action version $action list

it's already support in the backend, wsk work havn't started yet

• it is better to update the document

  • Change the create action flow's withDefaultVerstion -to withVersionParameter
  • Add the publish logic to the document

withDefaultVerstion is more suitable, it is used for users to set a default version for an action, and there is no publish logic now

[codecov-commenter]

Codecov Report

Merging #4986 (5a23339) into master (3ea756f) will increase coverage by 1.03%. The diff coverage is 57.20%.

:exclamation: Current head 5a23339 differs from pull request most recent head cc56908. Consider uploading reports for the commit cc56908 to get more accurate results

@@            Coverage Diff             @@
##           master    #4986      +/-   ##
==========================================
+ Coverage   38.23%   39.26%   +1.03%     
==========================================
  Files         240      240              
  Lines       14569    14728     +159     
  Branches      647      615      -32     
==========================================
+ Hits         5570     5783     +213     
+ Misses       8999     8945      -54     

Impacted Files	Coverage Δ
.../main/scala/org/apache/openwhisk/core/WarmUp.scala	0.00% <0.00%> (ø)
...ache/openwhisk/core/database/DocumentHandler.scala	0.00% <0.00%> (ø)
...nwhisk/core/database/RemoteCacheInvalidation.scala	78.12% <0.00%> (-2.53%)	:arrow_down:
...sk/core/database/cosmosdb/CosmosDBViewMapper.scala	0.00% <0.00%> (ø)
...nwhisk/core/database/memory/MemoryViewMapper.scala	0.00% <0.00%> (ø)
.../openwhisk/core/loadBalancer/FPCPoolBalancer.scala	0.00% <0.00%> (ø)
...k/core/containerpool/v2/InvokerHealthManager.scala	0.00% <ø> (ø)
...nwhisk/core/invoker/ContainerMessageConsumer.scala	0.00% <0.00%> (ø)
...rg/apache/openwhisk/core/controller/Packages.scala	72.38% <46.66%> (-1.46%)	:arrow_down:
...org/apache/openwhisk/core/controller/Actions.scala	70.34% <51.81%> (-11.83%)	:arrow_down:
... and 35 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[bdoyle0182]

Are there any open concerns with this PR? Since this is backwards compatible, I don't have any concerns with finally merging this to master once built. The only thing I can think of would be to wait until there's support for all action types, but I'm not sure if that's worth waiting for. This is a very important feature to keep up with other FaaS offerings.

[jiangpengcheng]

Are there any open concerns with this PR? Since this is backwards compatible, I don't have any concerns with finally merging this to master once built. The only thing I can think of would be to wait until there's support for all action types, but I'm not sure if that's worth waiting for. This is a very important feature to keep up with other FaaS offerings.

thanks for review, I'm waiting for all new scheduler prs get merged, and add versioning support for them, after then I will send an email to mail list for a discussion

[bdoyle0182]

Bumping this pr since all pr's for the new scheduler have been merged in. It would be great to this feature in now finally. It's a big deal to have

[jiangpengcheng]

@bdoyle0182 thanks for reminding, I will handle this after Lunar New Year vacation

[bdoyle0182]

It would be great to finally get this feature in now that the scheduler is open sourced.

One idea I had was how can we handle dev / snapshotting versions? Since right now it seems like it will only support strict semantic versioning of /d*/./d*/./d. Or if it's something we really need?

[bdoyle0182]

resolved conflicts

[bdoyle0182]

One Additional ToDo on top of what's listed in the description, add view mapper for actions-versions view to the MongoDb view mappers