lrs-conformance-test-suite
lrs-conformance-test-suite copied to clipboard
adlnet
xAPI 2.0 Compatibility + Test Directory
This update adds xAPI 2.0 compatibility to the conformance test suite.
- Version support for 2.0.x has been added.
- 2.0 test directory has been added with new test sets for Context Agents and Context Groups, as well as several improvements and adjustments to account for LRS deltas between 1.0.3 and 2.0.
@Argenrost @vbhayden You might want to look at the testing for XAPI-00321 (works in a grep), it looks like it still expects the old 1.0.x version for everything aside from the /statements resource, we get: AssertionError: expected '2.0.0' to match /^1\.0\.\d+$|^0?\.9\d*?$/ on /agents, /activities, etc.
Some tests also now seem to expect an X-Experience-Api-Version header on 400 responses (not sure they did before), couldn't find discussion of that in the changelogs.
The changes look good for setting the headers! I'm now running into an issue when I try to make an LRS that is backwards compatible with 1.0.x: It looks like there's a test that tries to ensure the alternate request syntax is not accepted, but does not itself send an X-Experience-Api-Version header for 2.0.0. Since the old alt request syntax requests encode that in the body, the LRS has no way to know it is a 2.0.0 request.
There was no XAPI-XXXX code for the fail but it did give me the loc+line: lrs-conformance-test-suite/test/v2_0/H.Communication1.3-AlternateRequestSyntax.js:108:14
Better late than never, hopefully fixed with https://github.com/adlnet/lrs-conformance-test-suite/pull/252/commits/46d4ee02d75fdea099343a442c965bee9d608d3f
Thanks Milt!
@vbhayden Looks like the issue I described is indeed fixed, however there are still some issues with making an LRS backwards compatible. Need to run them all down but for starters the test for XAPI-00316 will still fail when using 1.0.3 mode (via the --xapiVersion 1.0.3 arg you added). This is a test for the array of supported versions in the /about resource which looks like it will need to be updated to support backwards-compatible LRSs. This test is grep-able with -g XAPI-00316.
I guess that test could be updated yeah, the test does seem a bit aggressive against the actual reading of the original spec:
An LRS MUST return the JSON document described above, with a "version" property that includes the latest minor and patch version the LRS conforms to, for each major version.
For version 1.0.0 of this specification, this means that 1.0.0 MUST be included; 0.9 and 0.95 MAY be included. (For the purposes of this requirement, 0.9 and 0.95 are considered major versions.)
We've historically been a little shy about updating the 1.0.3 tests outside of weird behavior, but something like a new major version seems important enough.
Agreed. The way I read 2.8 it doesn't explicitly disallow adding versions to the list, just specifies what 1.0.3 expects to find there, and one would expect that as novel versions are supported they would appear in the list.
Just updated the 1.0.3 and 2.0.0 about tests to hopefully be cross-compatible with each other etc.
Just updated the 1.0.3 and 2.0.0 about tests to hopefully be cross-compatible with each other etc.
Thanks, I'll run my tests again!
➜ lrs-conformance-test-suite git:(master) ✗ node bin/console_runner.js \
-e http://localhost:8080/xapi \
-bz --xapiVersion 1.0.3
error: unknown option `--xapiVersion'
Did the option change since yesterday? These same args worked to run the 1.0.3 tests before.
➜ lrs-conformance-test-suite git:(master) ✗ node bin/console_runner.js \ -e http://localhost:8080/xapi \ -bz --xapiVersion 1.0.3 error: unknown option `--xapiVersion'Did the option change since yesterday? These same args worked to run the
1.0.3tests before.
My bad, wrong branch
➜ lrs-conformance-test-suite git:(LRS-2.0) node bin/console_runner.js \
-e http://localhost:8080/xapi \
-bz --xapiVersion 1.0.3
runTests starting
Options not valid ValidationError: child "directory" fails because ["directory" does not contain 1 required value(s)]
{"total":null,"passed":null,"failed":null}
Tests completed in 0 seconds
Full run log written to /Users/milt/projects/lrs-conformance-test-suite/logs/773ed60e-97f0-4709-9800-9423900a83d3.log
Closed
Hmm, something is still amiss, looks like I need to specify a directory now?
➜ lrs-conformance-test-suite git:(LRS-2.0) node bin/console_runner.js \
-e http://localhost:8080/xapi \
-bz --xapiVersion 1.0.3 --directory v1_0_3
runTests starting
Cannot specify both an xAPI Version and a Directory.
{"total":null,"passed":null,"failed":null}
Tests completed in 0 seconds
Full run log written to /Users/milt/projects/lrs-conformance-test-suite/logs/374387f7-9f6c-4424-ba7e-4efac83091eb.log
Closed
It doesn't seem to like that either, but when I omit the xapiVersion arg it will fail on the first test because the tests send the version header as undefined:
➜ lrs-conformance-test-suite git:(LRS-2.0) node bin/console_runner.js \
-e http://localhost:8080/xapi \
-bz --directory v1_0_3
runTests starting
Attempting xAPI Conformance Suite Against:
xAPI Version: undefined
Test Path(s): v1_0_3
LRS Endpoint: http://localhost:8080/xapi
Grep is undefined
Setting up
Accounting for time differential between test suite and lrs
{"total":1365,"passed":0,"failed":1,"version":"2.0.0"}
Tests completed in 0.159 seconds
Test Suite Complete
Full run log written to /Users/milt/projects/lrs-conformance-test-suite/logs/c3b085e4-e297-40e8-9293-7f2c6abb9890.log
Closed
Note that it also fails now with minimal args:
➜ lrs-conformance-test-suite git:(LRS-2.0) node bin/console_runner.js -e http://localhost:8080/xapi
runTests starting
Options not valid ValidationError: child "directory" fails because ["directory" does not contain 1 required value(s)]
{"total":null,"passed":null,"failed":null}
Tests completed in 0 seconds
Full run log written to /Users/milt/projects/lrs-conformance-test-suite/logs/92ccfb84-f2bb-4a36-bfd5-eb427c968d4f.log
Closed
So the setup atm should only require you to specify a version to test against, with directory another way of going about specifying tests. The integration with the conformance site seems ok atm, so something might've gotten overlooked for the cli portion.
I'll check that tonight.
Yeah, looks like it was requiring a directory to be specified in lieu of a version, but this never actually assigned the version later on etc. Should be able to infer the version from the directory now, although tbh I don't really like the assignment of directory as an arg in the first place and might discuss deprecating that in the future.
Thanks for the heads up!
Thanks Trey! That definitely resolved the issues with the runner.
Running the 1.0.3 tests now I'm seeing 4 remaining failures with our testing LRS that seem to assert that the response X-Experience-API-Version header should be 1.0.3 rather than the latest version, 2.0.0. Is the intention that an LRS, after determining that the request is specifying 1.0.3, also return that in the response? That's kind of what I draw from the spec when it says:
The LRS shall reject requests with version header prior to version 2.0.0 unless such requests are routed to a fully conformant implementation of a prior version specified in the header.
But I wanted to get your read on it.
If that is indeed the case then there are only 3 remaining tests that would prevent backwards-compatibility, falling under XAPI-00331. These tests assert that the response X-Experience-API-Version header should be 1.0.3 when no version header is provided in the request (these are 400 error responses for missing version), which makes it impossible for an LRS endpoint to be backwards compatible since it would have no way to know the old version is expected. Reproduction command:
node bin/console_runner.js -e http://localhost:8080/xapi --xapiVersion 1.0.3 -z -g XAPI-00331
My suggestion in these cases (400 for missing version header) is that the tests simply verify that the header is present, not that it is a specific version).
Tbh I'm not super sure where some of the interpretations came from that led to those specific tests. The spec didn't seem super aggressive about locking down to one version, but maybe there's a line somewhere that I'm not seeing.
I'm out today but might check through some this evening for sanity. 🤔
I think I can see the impetus in 1.0.3 here where it says:
The LRS MUST include the "X-Experience-API-Version" header in every response.
BUT yeah it kind of breaks versioning to test it specifically like that (on 400s for no version). Enjoy your time off!
Afternoon @milt,
We have a staging server up for these changes at https://lrstest.staging.adlnet.gov, so feel free to create an account there and see if things look OK on your end for your 2.0 updates.
I've also created an issue to track known / ongoing issues with the 2.0 suite over here: https://github.com/adlnet/lrs-conformance-test-suite/issues/258
Thanks for all of your help over the last year or so.
Happy to help, thanks for being so responsive! I logged in and made an account but we actually don't actually have anything running publicly that supports 2.0.0 to run against right now.
So far I've been testing with a branch of our LRS library against the local CLI. Right now that branch passes the 2.0 conformance tests (from the branch on this PR), but fails 3 of the 1.0.3 tests because of the backwards compatibility issue I described above. I'll link it in the issue you created for reference.
Hi Trey! Getting some weird nonsensical test output after the latest commits and was wondering if you had any idea what was going on. For example, here's an output where it appears to be failing a test for an ETAG requirement on documents but the only listed requests/responses pertain to the statements endpoint:
{
"name": "console",
"owner": null,
"flags": {
"endpoint": "http://localhost:8080/xapi"
},
"options": {},
"rollupRule": "mustPassAll",
"uuid": "4f291c65-e984-44d3-8bce-e2a7460bf264",
"startTime": 1685984406843,
"endTime": 1685984407328,
"duration": 485,
"state": "finished",
"summary": {
"total": 1408,
"passed": 28,
"failed": 1,
"version": "2.0.0.0"
},
"log": {
"title": "",
"name": "",
"requirement": "",
"log": "REQUEST SUPERREQUEST\n_______________________________________\nPOST /xapi/statements HTTP/1.1\r\nX-Experience-API-Version: 2.0.0\r\nhost: localhost:8080\r\naccept: application/json\r\ncontent-type: application/json\r\ncontent-length: 324\r\nConnection: close\r\n\r\n{\"actor\":{\"objectType\":\"Agent\",\"name\":\"xAPI mbox\",\"mbox\":\"mailto:[email protected]\"},\"verb\":{\"id\":\"http://adlnet.gov/expapi/verbs/attended\",\"display\":{\"en-GB\":\"attended\",\"en-US\":\"attended\"}},\"object\":{\"objectType\":\"Activity\",\"id\":\"http://www.example.com/meetings/occurances/34534\"},\"id\":\"b1c28439-90c8-4f2d-8db2-7455c9225e38\"}\n\nRESPONSE SUPERREQUEST\n_______________________________________\nHTTP/1.1 200 OK\nconnection: close\ndate: Mon, 05 Jun 2023 17:00:06 GMT\nx-experience-api-consistent-through: 2023-06-05T17:00:06.947459000Z\nx-experience-api-version: 2.0.0\ncontent-type: application/json;charset=utf-8\n\nb1c28439-90c8-4f2d-8db2-7455c9225e38\n=======================================\nREQUEST SUPERREQUEST\n_______________________________________\nGET /xapi/statements?statementId=b1c28439-90c8-4f2d-8db2-7455c9225e38 HTTP/1.1\r\nX-Experience-API-Version: 2.0.0\r\nhost: localhost:8080\r\nConnection: close\r\n\r\nRESPONSE SUPERREQUEST\n_______________________________________\nHTTP/1.1 200 OK\nconnection: close\ndate: Mon, 05 Jun 2023 17:00:06 GMT\nx-experience-api-consistent-through: 2023-06-05T17:00:06.970985000Z\nx-experience-api-version: 2.0.0\ncontent-type: application/json;charset=utf-8\n\n{\"actor\":{\"objectType\":\"Agent\",\"name\":\"xAPI mbox\",\"mbox\":\"mailto:[email protected]\"},\"verb\":{\"id\":\"http://adlnet.gov/expapi/verbs/attended\",\"display\":{\"en-GB\":\"attended\",\"en-US\":\"attended\"}},\"object\":{\"objectType\":\"Activity\",\"id\":\"http://www.example.com/meetings/occurances/34534\"},\"id\":\"b1c28439-90c8-4f2d-8db2-7455c9225e38\",\"stored\":\"2023-06-05T17:00:06.940355000Z\",\"timestamp\":\"2023-06-05T17:00:06.940355000Z\",\"authority\":{\"name\":\"Memory LRS\",\"objectType\":\"Agent\",\"account\":{\"name\":\"root\",\"homePage\":\"http://localhost:8080\"}},\"version\":\"2.0.0\"}\n=======================================\n",
"status": "failed",
"tests": [
{
"title": "(4.2.7) Concurrency",
"name": "",
"requirement": "4.2.7",
"log": "",
"status": "failed",
"tests": [
{
"title": "xAPI uses HTTP 1.1 entity tags (ETags) to implement optimistic concurrency control in the following resources, where PUT, POST or DELETE are allowed to overwrite or remove existing data.",
"name": "xAPI uses HTTP 1.1 entity tags (ETags) to implement optimistic concurrency control in the following resources, where PUT, POST or DELETE are allowed to overwrite or remove existing data.",
"requirement": "",
"log": "",
"status": "failed",
"tests": [
{
"title": "Concurrency for the Activity State Resource.",
"name": "Concurrency for the Activity State Resource.",
"requirement": "",
"log": "",
"status": "failed",
"tests": [
{
"title": "If a PUT request is received without either header for a resource that already exists",
"name": "If a PUT request is received without either header for a resource that already exists",
"requirement": "",
"log": "",
"status": "failed",
"tests": [
{
"title": "Return 409 conflict",
"name": "Return 409 conflict",
"requirement": "",
"log": "",
"status": "failed",
"error": "AssertionError: expected 204 to equal 409",
"tests": []
}
]
}
]
}
]
}
]
}
]
}
}
I'm adding some extra instrumentation to my test LRS to see if I can figure out more, but let me know if this makes any sense to you.
Morning @milt, just now seeing this -- been a long summer.
There were issues with the ETag tests and timing due to the introduction of a few async calls being attributed to the wrong requirement. They should all be fine now, but let me know if they're still misbehaving for you.