FHIR icon indicating copy to clipboard operation
FHIR copied to clipboard

Published 20 hours ago • verified publisher icon LinuxForHealth

Integration test com.ibm.fhir.server.test.examples needs to be updated to monitor for error messages

Open d0roppe opened this issue 3 years ago • 3 comments

Is your feature request related to a problem? Please describe. In running this integration test com.ibm.fhir.server.test.examples, if it gets some errors it should be more aware of that and fail if they occur

	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:836)
Feb 07, 2022 3:05:04 PM com.ibm.fhir.model.spec.test.R4ExamplesDriver processIndex
INFO: Overall success rate = 16/629 = 2%. Took 130415 ms
PASSED: processExamples

===============================================
    Default test
    Tests run: 1, Failures: 0, Skips: 0
===============================================


===============================================
Default suite
Total tests run: 1, Passes: 1, Failures: 0, Skips: 0
===============================================

Describe the solution you'd like if the test encounters error messages in data setup, it should fail

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Acceptance Criteria

Run the test and if errors occurred it should not pass.

Additional context Add any other context or screenshots about the feature request here.

d0roppe avatar Feb 08 '22 11:02 d0roppe

Seeing this again:

WARNING: Response body: {"resourceType":"OperationOutcome","id":"ac-12-0-4-6e13de61-83fb-429b-8a14-bd53354aa837","issue":[{"severity":"error","code":"not-found","details":{"text":"The requested resource type 'Evidence' is not found"}}]}
Exception in thread "pool-2-thread-5" java.lang.AssertionError: expected:<201> but was:<404>
	at org.testng.AssertJUnit.fail(AssertJUnit.java:65)
	at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:467)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:88)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:318)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:328)
	at com.ibm.fhir.server.test.FHIRServerTestBase.assertResponse(FHIRServerTestBase.java:588)
	at com.ibm.fhir.server.test.examples.ExampleRequestProcessor.process(ExampleRequestProcessor.java:86)
	at com.ibm.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:375)
	at com.ibm.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:272)
	at com.ibm.fhir.model.spec.test.R4ExamplesDriver$1.run(R4ExamplesDriver.java:204)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Aug 01, 2022 10:05:39 PM com.ibm.fhir.server.test.examples.ExampleRequestProcessor process
WARNING: Unexpected response for JSON file: json/ibm/complete-mock/EvidenceVariable-1.json
Aug 01, 2022 10:05:39 PM com.ibm.fhir.server.test.examples.ExampleRequestProcessor process
WARNING: Response body: {"resourceType":"OperationOutcome","id":"ac-12-0-4-8e1a9733-a339-4136-a60f-2d4533007ed8","issue":[{"severity":"error","code":"not-found","details":{"text":"The requested resource type 'EvidenceVariable' is not found"}}]}
Exception in thread "pool-2-thread-6" java.lang.AssertionError: expected:<201> but was:<404>
	at org.testng.AssertJUnit.fail(AssertJUnit.java:65)
	at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:467)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:88)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:318)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:328)
	at com.ibm.fhir.server.test.FHIRServerTestBase.assertResponse(FHIRServerTestBase.java:588)
	at com.ibm.fhir.server.test.examples.ExampleRequestProcessor.process(ExampleRequestProcessor.java:86)
	at com.ibm.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:375)
	at com.ibm.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:272)
	at com.ibm.fhir.model.spec.test.R4ExamplesDriver$1.run(R4ExamplesDriver.java:204)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Aug 01, 2022 10:06:06 PM com.ibm.fhir.server.test.examples.ExampleRequestProcessor process
WARNING: Unexpected response for JSON file: json/ibm/complete-absent/SubstanceDefinition-1.json
Aug 01, 2022 10:06:06 PM com.ibm.fhir.server.test.examples.ExampleRequestProcessor process
WARNING: Response body: {"resourceType":"OperationOutcome","id":"ac-12-0-4-769f0b9d-62d0-4d63-aa1a-2b6eea68e234","issue":[{"severity":"error","code":"not-found","details":{"text":"The requested resource type 'SubstanceDefinition' is not found"}}]}
Exception in thread "pool-2-thread-3" java.lang.AssertionError: expected:<201> but was:<404>
	at org.testng.AssertJUnit.fail(AssertJUnit.java:65)
	at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:467)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:88)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:318)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:328)
	at com.ibm.fhir.server.test.FHIRServerTestBase.assertResponse(FHIRServerTestBase.java:588)
	at com.ibm.fhir.server.test.examples.ExampleRequestProcessor.process(ExampleRequestProcessor.java:86)
	at com.ibm.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:375)
	at com.ibm.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:272)
	at com.ibm.fhir.model.spec.test.R4ExamplesDriver$1.run(R4ExamplesDriver.java:204)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Aug 01, 2022 10:06:07 PM com.ibm.fhir.model.spec.test.R4ExamplesDriver waitForCompletion
INFO: Waiting for all requests to complete (remaining = 4)
Aug 01, 2022 10:06:08 PM com.ibm.fhir.model.spec.test.R4ExamplesDriver waitForCompletion
INFO: All requests complete
Aug 01, 2022 10:06:08 PM com.ibm.fhir.model.spec.test.R4ExamplesDriver processIndex
INFO: Overall success rate = 460/463 = 99%. Took 85976 ms
]]>

lmsurpre avatar Aug 02 '22 02:08 lmsurpre

After the fix, the test was failing 3 cases and it properly failed the test. After fixing that, it passed again.

lmsurpre avatar Aug 04 '22 13:08 lmsurpre

hmm, there must still be some way that we're not picking up the errors because in a recent test I noted the following output:

INFO: Processing index 'MINIMAL_JSON'
Error: Exception in thread "pool-2-thread-1" java.lang.AssertionError: expected:<200> but was:<404>
	at org.testng.AssertJUnit.fail(AssertJUnit.java:65)
	at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:467)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:88)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:318)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:328)
	at org.linuxforhealth.fhir.server.test.FHIRServerTestBase.assertResponse(FHIRServerTestBase.java:588)
	at org.linuxforhealth.fhir.server.test.examples.ExampleRequestProcessor.process(ExampleRequestProcessor.java:108)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:374)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:268)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver$1.run(R4ExamplesDriver.java:211)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Error: Exception in thread "pool-2-thread-4" java.lang.AssertionError: expected:<200> but was:<404>
	at org.testng.AssertJUnit.fail(AssertJUnit.java:65)
	at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:467)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:88)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:318)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:328)
	at org.linuxforhealth.fhir.server.test.FHIRServerTestBase.assertResponse(FHIRServerTestBase.java:588)
	at org.linuxforhealth.fhir.server.test.examples.ExampleRequestProcessor.process(ExampleRequestProcessor.java:108)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:374)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:268)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver$1.run(R4ExamplesDriver.java:211)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Error: Exception in thread "pool-2-thread-6" java.lang.AssertionError: expected:<200> but was:<404>
	at org.testng.AssertJUnit.fail(AssertJUnit.java:65)
	at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:467)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:88)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:318)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:328)
	at org.linuxforhealth.fhir.server.test.FHIRServerTestBase.assertResponse(FHIRServerTestBase.java:588)
	at org.linuxforhealth.fhir.server.test.examples.ExampleRequestProcessor.process(ExampleRequestProcessor.java:108)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:374)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:268)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver$1.run(R4ExamplesDriver.java:211)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Aug 11, 2022 3:36:31 PM org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver waitForCompletion
INFO: Waiting for all requests to complete (remaining = 5)
Aug 11, 2022 3:36:32 PM org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver waitForCompletion
INFO: All requests complete
Aug 11, 2022 3:36:32 PM org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver processIndex
INFO: Overall success rate = 460/463 = 99%. Took 82351 ms

lmsurpre avatar Aug 11 '22 17:08 lmsurpre

Options:

A. update fhir-server-config to prevent create interactions for one or more resource types. this should result in 4xx errors for the corresponding example creates. this should fail the tests

B. alternatively, just change the client type to fhirVersion 4.0 in the tests...that was basically the original error that I was seeing

lmsurpre avatar Aug 30 '22 13:08 lmsurpre

QA:

Modified the fhir-server-config.json to disable Patient CREATE interactions:

       "resources": {
            "open": true,
            "Patient": {
                "interactions": ["read", "vread", "history", "search"]
            }
        },

Running the R4ExamplesServerTest we see the creation error being handled appropriately:

Aug 30, 2022 3:00:22 PM org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver processIndex
INFO: Overall success rate = 459/463 = 99%. Took 57851 ms
Aug 30, 2022 3:00:22 PM org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver processIndex
WARNING: json/spec/patient-glossy-example.json: wanted:         OK got:    PROCESS
Aug 30, 2022 3:00:22 PM org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver processIndex
WARNING: json/spec/patient-example.json: wanted:         OK got:    PROCESS
Aug 30, 2022 3:00:22 PM org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver processIndex
WARNING: json/complete-mock/Patient-1.json: wanted:         OK got:      PARSE
Aug 30, 2022 3:00:22 PM org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver processIndex
WARNING: json/complete-mock/Patient-2.json: wanted:         OK got:    PROCESS
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 59.541 s <<< FAILURE! - in org.linuxforhealth.fhir.server.test.examples.R4ExampleServerTest
[ERROR] org.linuxforhealth.fhir.server.test.examples.R4ExampleServerTest.processExamples  Time elapsed: 57.889 s  <<< FAILURE!
java.lang.Exception: Unexpected response for JSON file: json/spec/patient-glossy-example.json
	at org.linuxforhealth.fhir.server.test.examples.ExampleRequestProcessor.process(ExampleRequestProcessor.java:105)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:374)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:268)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver$1.run(R4ExamplesDriver.java:211)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.AssertionError: expected:<201> but was:<400>
	at org.testng.AssertJUnit.fail(AssertJUnit.java:65)
	at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:467)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:88)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:318)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:328)
	at org.linuxforhealth.fhir.server.test.FHIRServerTestBase.assertResponse(FHIRServerTestBase.java:588)
	at org.linuxforhealth.fhir.server.test.examples.ExampleRequestProcessor.process(ExampleRequestProcessor.java:100)
	... 6 more

Then by allowing create/update but not read, we see a similar issue (albeit handled in a different block):

        "resources": {
            "open": true,
            "Patient": {
                "interactions": ["create", "update", "history", "search"]
            }
        },

producing:

Aug 30, 2022 3:05:03 PM org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver processIndex
INFO: Overall success rate = 459/463 = 99%. Took 73457 ms
Aug 30, 2022 3:05:03 PM org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver processIndex
WARNING: json/spec/patient-example.json: wanted:         OK got:    PROCESS
Aug 30, 2022 3:05:03 PM org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver processIndex
WARNING: json/spec/patient-glossy-example.json: wanted:         OK got:    PROCESS
Aug 30, 2022 3:05:03 PM org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver processIndex
WARNING: json/complete-mock/Patient-1.json: wanted:         OK got:      PARSE
Aug 30, 2022 3:05:03 PM org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver processIndex
WARNING: json/complete-mock/Patient-2.json: wanted:         OK got:    PROCESS
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 75.112 s <<< FAILURE! - in org.linuxforhealth.fhir.server.test.examples.R4ExampleServerTest
[ERROR] org.linuxforhealth.fhir.server.test.examples.R4ExampleServerTest.processExamples  Time elapsed: 73.496 s  <<< FAILURE!
java.lang.Exception: Unexpected response for JSON file: json/spec/patient-example.json
	at org.linuxforhealth.fhir.server.test.examples.ExampleRequestProcessor.process(ExampleRequestProcessor.java:128)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:374)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:268)
	at org.linuxforhealth.fhir.model.spec.test.R4ExamplesDriver$1.run(R4ExamplesDriver.java:211)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.AssertionError: expected:<200> but was:<400>
	at org.testng.AssertJUnit.fail(AssertJUnit.java:65)
	at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:467)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:88)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:318)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:328)
	at org.linuxforhealth.fhir.server.test.FHIRServerTestBase.assertResponse(FHIRServerTestBase.java:588)
	at org.linuxforhealth.fhir.server.test.examples.ExampleRequestProcessor.process(ExampleRequestProcessor.java:123)
	... 6 more

Behaving as expected.

punktilious avatar Aug 30 '22 14:08 punktilious