FDC3 Software Conformance Framework

[ColinEberhardt]

Hello to all FDC3 users and maintainers, I’d like to propose the creation of an FDC3 Software Conformance framework, and am eager to get your feedback.

FDC3 has become a mature and valuable open standard for applications running on the financial desktop. The standard is still under active development, with a number of candidate features identified for version 2.0. As the standard expands in scope, and the number of implementations increase, the consumers of these implementations would benefit from a formal indication of compliance - i.e. that a product, application or service faithfully implements a specific FDC3 version.

Furthermore, the move to a more formal approach to compliance would help resolve any ambiguities in the API specification, or the current outlined approach to compliance.

Our goal is to create an automated test suite that ensures all aspects of the API specification are correctly implemented. This suite may be provided as a web-application, or it may be an automation suite appropriate for use within CI pipelines.

We’ll work with the FINOS / Linux Foundation teams and the FDC3 community to determine how this test suite, plus any necessary associated terms for conformance, are governed. For reference, a very mature example in the Linux Foundation is the Kubernetes conformance test suite, owned by the Community, which backs the CNCF conformance program, owned by the CNCF.

I’m interested in your thoughts and feedback on the above? I’m sure there are technical concerns to consider, and that this group will no doubt have lots of ideas around how these may be addressed (and I’d love to enter into discussion). But in the first instance, can I ask you to respond with an indication whether you are supportive of this concept?

[mindthegab]

From a FINOS standpoint, we'd be supportive to build a conformance program if the FDC3 project maintainers are supportive of this proposal for an open source conformance testing framework - this is in line with what was discussed several times at the Governing Board meeting over the last 2/3 quarters, as a reflection of the growing importance of this standard and to make sure out of the box interoperability for products claiming compatibility with the FDC3 standard.

Please let us know!

[kriswest]

Hi Colin,

The FDC3 maintainers have previously been consulted on support for a conformance framework/compliance testing, and responded positively (and for my part I can confirm that I am strongly in favour). At that time we also highlighted some of the ambiguities in both the API specification and compliance documentation which make the development of such formal testing difficult. This resulted in the 'formal specification' project that we've undertaken, with the help of FINOS and Rex Jaeschke. You can see the issues that have been raised by that project here , many of which have resulted in changes that are now merged into the pre-draft of FDC3 2.0 (master branch of the github repo, and visible on the web here).

Of particular interest are two of the remaining issues, which I'm about to start work on resolving:

• Compliance #484
  • Seeks to make compliance information explicit within the specification and reference material
• Reorganizing the Specification and Incorporating Reference Material #491
  • Reorganizes the standard material under a more consistent heading and formally incorporates the reference docs into the standard (which will contain explicit compliance information post #484), instead of the current weak reference to the typescript sources.

It is my hope and intention that resolving these issues will make it much easier for both 'platform providers' (those implementing a Desktop Agent) to produce a compliant implementation and for a conformance testing framework to be implemented - hence, I would be happy to involve any team working on conformance testing in the generation or review any PRs associated with these issues.

At present, I believe this can be achieved without changing compliance requirements for any existing features, although we may need to revisit some of the accepted 2.0 proposals to confirm if any should be optional. As part of that work, I also intend to raise an issue to add support for feature flags to the ImplementationMetadataobject that is retrieved via fdc3.getInfo() which will enable implementations to discover which optional features are available in a particular implementation. It should be noted that if optional features are supported they should conform to their definitions in the standard.

N.B. Throughout this response, I am assuming that your proposed work will focus on compliance for 'Platform Providers' (i.e. Desktop Agent implementations) - compliance for Application providers is currently much looser, with the majority of requirements being recommended rather than required (i.e. they are labelled with SHOULD, rather than MUST). There may be a separate conversation to be had about compliance for application providers.

I look forward to discussing this work further.

[mindthegab]

With support of @finos/fdc3-maintainers, the work on building a software conformance framework has started (see first iteration).

I suggest we keep this issue open (@ColinEberhardt i've assigned to you) until the first milestone is completed and framework is functional.

[mindthegab]

Hey @ColinEberhardt @kriswest - given tests are now going to be in a separate repo, should this issue be closed now as well as the related #615 ?

[kriswest]

Are they going to stay in a separate repo, or be contributed forwards in the fdc3 repo (toolbox folder) when complete?

[robmoffat]

FDC3 is already a multi-repo project within FINOS so I was planning on keeping it as a separate repo, unless there's a good reason not to...?

[kriswest]

unless there's a good reason not to...?

I thought hosting that's consistent with FDC3 Workbench and FDC3 Explained in the toolbox folder would be good and saves on a separate hosting/deployment setup/subdomain. Also, it might be easier to maintain alongside changes to the standard if it's in the same repo (i.e. you can tweak the API types and the relevant conformance test in the same PR) - once it reaches parity with the standard that is.

It'll probably need duplicating for each version to host multiple copies at the same time and should probably only go in when it's accepted as complete for at least the first version anyway - so development on a separate repo has its advantages for the time being.

[ColinEberhardt]

Hi @kriswest I am not at all opposed to it folding into this repo when the time is right. However, at the moment, when we are looking to iterate on it a bit more rapidly, it made sense to do it in a separate repo.

[mindthegab]

I might be missing something, but I don't think it makes sense to fold everything in the same repo. While ultimately the FDC3 maintainers should do what they feel best for the FDC3 project itself, I feel strongly the conformance framework should a separate repo because:

1. This is a different set of maintainers (@ColinEberhardt is maintaining this) - Github doesn't offer "sub repo" permissioning, so effectively either we add @ColinEberhardt as FDC3 maintainer for the whole repo or he will not have write access. I think using subfolders for this purpose is just wrong and counter to any open source project I know.

2. The conformance framework will likely follow a different release lifecycle - ie. it will come AFTER the standard itself is released. Unless you are suggesting the FDC3 maintainers team plans to update the conformance framework at every release BEFORE releasing the standard. For my experience basing a separate component in a "sub folder" vs a proper git repo (the atom of collaboration) creates only more complexity/additional confibguration in build and release, as build tools generally expect you to be at the based of the repo

And while point 2 can be worked around, I think point 1 is really an important one - yes, I would surely trust @ColinEberhardt not to touch anything beyond his subfolder, but really "trust" is not a great scalable way to enforce ACLs.

For an example, we're modeling against, check out the Kubernetes conformance tests in a dedicated repo.

If you really feel you want to show everything in the FDC3 repo (and I argue tho that our "consistency" aggregator is the microsite, not the repo), I think you should really consider git submodules, where we can still have separate repos with separate permissions and lifecycles, but pull them together through what effectively are symlinks. In fact I'd argue the toolbox and FDC3 explained should also be submodules - so they could have a life of their own while still being included in the main FDC3 repo as needed.

[kriswest]

I might be missing something, but I don't think it makes sense to fold everything in the same repo. While ultimately the FDC3 maintainers should do what they feel best for the FDC3 project itself, I feel strongly the conformance framework should a separate repo because:

1. This is a different set of maintainers @.*** https://github.com/ColinEberhardt is maintaining this) - Github doesn't offer "sub repo" permissioning, so effectively either we add @ColinEberhardt https://github.com/ColinEberhardt as FDC3 maintainer for the whole repo or he will not have write access.

I think it very odd for a set of conformance tests for a standard to be under different governance to the standard its testing conformance with. Ultimately, it needs to be accepted as correct to have standing and that acceptance should be under the usual review and governance of the standard. It would also make sense for the wider community to participate in maintenance.

You don't need write access to raise a PR, only to merge it.

I think using subfolders for this purpose is just wrong and counter to any

open source project I know.

I don't think mono-repos are all that uncommon in opensource projects.

2. The conformance framework will likely follow a different release

lifecycle - ie. it will come AFTER the standard itself is released. Unless you are suggesting the FDC3 maintainers team plans to update the conformance framework at every release BEFORE releasing the standard.

Eventually (i.e. its once its caught up-to-date) it would absolutely make sense to update the conformance tests at the same time as the standard draft - in fact it would help clarify the exact meaning and intent of changes and help with implementation of revisions to the standard - absolutely what we should be aiming for! We do already have tests in the FDC3 repo for the NPM module code and that is already updated in-sync with the standard. They currently have similar coverage to the conformance tests

• although the conformance tests need to grow significantly beyond them.

For my experience basing a separate component in a "sub folder" vs a proper git repo (the atom of collaboration) creates only more complexity/additional confibguration in build and release, as build tools generally expect you to be at the based of the repo

Which build tools? Build tools are generally set up to run from a base folder, with little interaction with the actual repository. For example, in the FDC3 repo you build the NPM module/linting/typescript type generation from the root, FDC3 Workbench from \toolbox\fdc3-workbench and the website from \website. They happily have different linting setups, dependencies, builds etc.. The website build is set up to rather simply run the other builds, do some work on the schemas and assemble the whole thing for hosting on fdc3.finos.org.

And while point 2 can be worked around, I think point 1 is really an important one - yes, I would surely trust @ColinEberhardt https://github.com/ColinEberhardt not to touch anything beyond his subfolder, but really "trust" is not a great scalable way to enforce ACLs.

I think you're worrying too much about whether they can merge whenever they want - and IMHO leaning the wrong way. Like all PRs, from anyone other than a maintainer, they can fork the repository and work there totally unhindered, raising a PR to merge the result back into FDC3. This was the original approach as the PR was raised from https://github.com/ScottLogic/FDC3. They don't require reviews to make changes to their fork, only to integrate it into FDC3. A delay in reviewing does not block anything other than integration into the FINOS FDC3 repo - or work they specifically need to be advised on how to complete/interpretation of the FDC3 spec (which is hopefully significantly helped by the formal spec project and the new compliance info round up on this PR: https://github.com/finos/FDC3/pull/753).

For an example, we're modeling against, check out the Kubernetes

conformance tests https://github.com/cncf/k8s-conformance in a dedicated repo.

That is a very different shaped community with many thousands of contributors, and Kubernetes itself living in a totally different github organisation to that conformance framework. What fits them is not likely to fit us the same way... Also Kubernetes is a software framework rather than a standard... their conformance framework is perhaps acting more like the standard and ensuring that the implementation isn't accidentally migrating off spec in releases. Our situation is different in that we are a standard, with implementations produced by third parties that need testing for conformance with it - hence you might argue that keeping the standard and confirmance testing together is More (not less) like what kubernetes/CNCF is doing. I'm no expert on that community however and am happy to be corrected on any of that by someone closer to it.

If you really feel you want to show everything in the FDC3 repo (and I

argue tho that our "consistency" aggregator is the microsite https://fdc3.finos.org/, not the repo), I think you should really consider git submodules https://gist.github.com/gitaarik/8735255, where we can still have separate repos with separate permissions and lifecycles, but pull them together through what effectively are symlinks.

The microsite is of course part of the repo and actually integrates teh builds of the sub-projects. There are a variety of mechanisms available for us to integrate - but what I'm concerned about is governance. I appreciate the work you put into helping this community to succeed (and this project is or will be, I hope, an important part of that). But I also don't think the community actually knows how this project came about, wasn't on how it would be implemented or governed, or on changes to that governance. For it to help, it needs to have standing in the community, needs to be trusted and needs to be maintained - likely with the help and consent of those implementing the standard as platform providers. At the bare minimum you need to involve those who will be tested by the framework and those who will benefit from the testing (the app providers) in its governance.

I'm not that bothered whether its a subfolder or a submodule (although In our experience submodules can be an unholy nightmare that always seem to result in moving to a mono repo) - but I do want to see it closely tied to the standard and at a particular revision with a formal acceptance process by the FDC3 community.

Finally, I want to add that I really want this project to succeed and appreciate the work you, FINOS and ScottLogic have done to bring it about. For it to have the effect I think we're all looking for I think it needs to be brought into the core FDC3 and adopted officially at key milestones. I look forward to the day that it is released on the same day that a standard version is adopted AND I look forward to contributing to making that happen ;-)

K

-- Kris West Principal Engineer [image: Cosaic] +1 (800) 821-8147 @.*** Cosaic.io: https://cosaic.io/ ChartIQ http://cosaic.io/ChartIQ + Finsemble http://cosaic.io/Finsemble

[rikoe]

@mindthegab @robmoffat I want to second @kriswest here. I think a discussion should happen between FDC3 maintainers and proposed conformance project maintainers to go through the pros and cons and define the relationship better.

[rikoe]

Regarding code ownership in a single repo, this is supported by GitHub today: https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners

[mindthegab]

@mindthegab @robmoffat I want to second @kriswest here. I think a discussion should happen between FDC3 maintainers and proposed conformance project maintainers to go through the pros and cons and define the relationship better.

I thought this already happened (at least with @kriswest and @robmoffat ) and there was an agreement to open a PR to @finos/fdc3-maintainers to bless "releases" of the conformance framework, as @finos/conformance-fdc3-maintainers need approval from @finos/fdc3-maintainers to ensure a certain release of the framework indeed correctly proves compliance to a certain version of FDC3. Basically, regardless of the underlying repo, everyone agrees the authority of blessing the conformance framework functional validity lies with @finos/fdc3-maintainers.

But ultimately guys, it's your project and you should decide what's best. I shared my, maybe outdated, views, but please do feel free to organize the project as you prefer.

[kriswest]

@mindthegab I haven't had any such discussions. Lets pick up this discussion on the fdc3-maintainers list.

[kriswest]

Closing this issue as the first iteration of the conformance framework was implemented, DA implementations tested and badges issued!