Known 2.0 Test Suite Issues

[vbhayden]

As we work through adding the 2.0 branch for this test suite in alignment with https://opensource.ieee.org/xapi/xapi-base-standard-documentation, I'll try to keep this issue updated with known issues / gap in the test suite.

Known Issues

• Not all tests will have verbose failure messages
• Links for the 2.0 tests still point to the adlnet/xAPI-Spec repo
• Appearance of tests may be unintuitive for the 2.0 branch

Discussion Topics

• Backwards compatibility issues due to the requirement of potentially incompatible xAPI version response header between 1.0.3 and 2.0.0.

Additionally, we have a staging server up to test these changes at https://lrstest.staging.adlnet.gov, so feel free to test the suite against a 2.0-compatible endpoint and report any issues here. Any findings would greatly help the team. 😊

Changes to LRS Requirements for xAPI 2.0.pdf

[milt]

Linking the version header issue here so it doesn't get lost: https://github.com/adlnet/lrs-conformance-test-suite/pull/252#issuecomment-1508742585

Essentially the 1.0.3 tests (as they are on the 2.0.0 branch) prevent an LRS from being compatible with both versions 1.0.3 and 2.0.0 as long as they require that an LRS return an X-Experience-API-Version header in a 400 response when no version header is present in the request. This means that an LRS cannot route requests to an appropriate backend and handle both versions which would be the best case scenario.

[vbhayden]

As an update, I was planning to deploy these updates today, along with the corresponding changes to the CTS and Adopter Registry sites.

For anyone worried that their LRS will suddenly be marked as out of date, its appearance shouldn't change at all etc. For example, the Watershed LRS's product page on the updated testing site branch still shows conformance to the current 1.0.3 version of the test suite:

They have been designed so that all current functionality with xAPI 1.0.3 will be preserved -- with the 2.0 options / conformance being available.

Also, I've updated the "Known Issues" portion of this ticket, with the big ones all being traceability for the 2.0 tests. This is due to the previous utilization of old / locally maintained versions of super-request and supertest, which has been used sparingly for the newer 2.0 tests due to how unpleasant the syntax is -- especially when frontloading multiple documents for a single test.

I will be making backups for each live system in case anything seems off (so the CTS site, the Adopter Registry, and the LRS), which is usually guaranteed when standing something up before a 3-day weekend etc.

If anyone has concerns or would have a reason to postpone this, let me know!

[vbhayden]

With the 2.0 branch being merged into master, I'll try to keep this card updated with ongoing issues. The only outstanding ones are still error verbosity and the links pointing to the wrong repo.

[vbhayden]

@milt For the backwards compatibility topic that came up on today's call, I am not sure of a way to allow it without updating the 1.0.3 spec -- if that's something that the community is fine with, then I'd have no issue with updating the 1.0.3 tests though?

The segment here: https://github.com/adlnet/xAPI-Spec/blob/master/xAPI-Communication.md#versioning

The LRS MUST reject requests with a version header of 1.1.0 or greater.

We could just add a line specifying that 2.X.Y would be acceptable for backwards compatibility.

[vbhayden]

The alternate request syntax is another thing, but I don't believe 2.0 specifically forbids it -- which may be overly prescriptive in how the CTS currently works iirc.