EMResearch
Only one testcase was generated for 1 API (possible issue with string constraints)
Although running more than 1 million tests, only 1 testcase was generated for 1 restful API.
Here is the command, using the default config. Is there any problem? I remember that it used to be possible to generate multiple testcases.
java -jar ~/Downloads/evomaster.jar --maxTime 7200s --outputFolder "/Users/apple/gitrepo/EvoMaster/whitebox-demo/src/main/java/long9"
*
_____ ___ ___ _
| ___| | \/ | | |
| |____ _____ | . . | __ _ ___| |_ ___ _ __
| __\ \ / / _ \| |\/| |/ _` / __| __/ _ \ '__|
| |___\ V / (_) | | | | (_| \__ \ || __/ |
\____/ \_/ \___/\_| |_/\__,_|___/\__\___|_|
* EvoMaster version: 3.4.0
* Loading configuration file from: /Users/apple/gitrepo/em.yaml
* Initializing...
00:28:14.270 [main] WARN o.t.u.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/Users/apple/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /Users/apple/.testcontainers.properties (No such file or directory)
* There is only 1 usable RESTful API endpoint defined in the schema configuration
* Starting to generate test cases
* Consumed search budget: 74.601%
* Consumed search budget: 99.994%
* Covered targets: 22; time per test: 4.0ms (1.0 actions); since last improvement: 4806s
* Starting to apply minimization phase
* Recomputing full coverage for 2 tests
02:28:15.179 [main] WARN o.e.core.search.service.Minimizer - Different status code 500 returned from original null, ie, null != 500 for endpoint: POST:/api/bind_card_apply
* Recomputing coverage did lose many targets, more than the threshold 20.0%: from 22 to 7, i.e., lost 15
* Missing targets:
* Line_at_com.example.demo.controller.DemoController_00034
* Success_Call_at_com.example.demo.controller.DemoController_00034_0
* Success_Call_at_com.example.demo.controller.DemoController_00034_1
* Success_Call_at_com.example.demo.controller.DemoController_00034_2
* Success_Call_at_com.example.demo.controller.DemoController_00034_3
* Success_Call_at_com.example.demo.controller.DemoController_00034_4
* Success_Call_at_com.example.demo.controller.DemoController_00034_5
* Line_at_com.example.demo.controller.DemoController_00036
* Success_Call_at_com.example.demo.controller.DemoController_00036_0
* Line_at_com.example.demo.controller.DemoController_00037
* Line_at_com.example.demo.controller.DemoController_00038
* Success_Call_at_com.example.demo.controller.DemoController_00038_0
* Success_Call_at_com.example.demo.controller.DemoController_00038_1
* -1:POST:/api/bind_card_apply
* RESPONSE_BODY_PAYLOAD_POST/api/bind_card_apply_null
* No test to minimize
* Minimization phase took 0 seconds
* Evaluated tests: 1188138
* Evaluated actions: 1307316
* Needed budget: 34%
* Passed time (seconds): 7200
* Execution time per test (ms): Avg=4.77 , min=2.00 , max=35173.00
* Execution time per action (ms): Avg=4.42 , min=2.00 , max=35173.00
* Computation overhead between tests (ms): Avg=1.29 , min=0.00 , max=250.00
* Computation overhead of resetting the SUT (ms): Avg=0.99 , min=0.00 , max=28983.00
* Computation overhead of fetching test results, per test, subset of targets (ms): Avg=0.56 , min=0.00 , max=5131.00
* Going to save 1 test to /Users/apple/gitrepo/EvoMaster/whitebox-demo/src/main/java/long9
* TCP timeouts: 2
* Potential faults: 2
* Covered targets (lines, branches, faults, etc.): 10
* Bytecode line coverage: 2% (1 out of 42 in 1 units/classes)
* Successfully executed (HTTP code 2xx) 0 endpoints out of 1 (0%)
* EvoMaster process has completed successfully
And, below is the api doc:
openapi: 3.1.0
info:
title: OpenAPI definition
version: v0
servers:
- url: http://localhost:8000
description: Generated server url
paths:
/api/bind_card_apply:
post:
tags:
- demo-controller
operationId: bindCardApply
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/CommonReqBindCardReq'
required: true
responses:
"200":
description: OK
content:
'*/*':
schema:
$ref: '#/components/schemas/CommonRespBindCardResp'
components:
schemas:
BindCardReq:
required:
- bankCardNo
- bankCardPhoneNumber
- idCardNo
- loanPersonName
type: object
properties:
idCardNo:
maxLength: 18
minLength: 18
pattern: "^[0-9]{17}[0-9Xx]$"
type: string
loanPersonName:
maxLength: 20
minLength: 0
type: string
bankCardNo:
maxLength: 19
minLength: 16
pattern: "^[0-9]+$"
type: string
bankCardPhoneNumber:
maxLength: 11
minLength: 11
pattern: "^[1][3-9][0-9]{9}$"
type: string
CommonReqBindCardReq:
type: object
properties:
appId:
type: string
data:
type: string
requestId:
type: string
timestamp:
type: string
key:
type: string
sign:
type: string
bizData:
$ref: '#/components/schemas/BindCardReq'
BindCardResp:
type: object
properties:
code:
type: string
msg:
type: string
sessionId:
type: string
CommonRespBindCardResp:
type: object
properties:
data:
type: string
key:
type: string
sign:
type: string
bizData:
$ref: '#/components/schemas/BindCardResp'
hi @ytfrank ,
thanks for reporting this issue. I think I know what is the problem. Probably it is related to:
bankCardNo:
maxLength: 19
minLength: 16
pattern: "^[0-9]+$"
we can handle constraints on length, and constraints on patterns, but not both at same time :(
The pattern one would prevail.
This needs to be fixed in EM.
For the time being, can you try to use pattern: "^[0-9]{16,19}$" and see if it helps?
Dear @ytfrank ,
thanks for trying out EvoMaster and opening an issue here. If you have 10 minutes, we would be grateful if you could fill this survey on test naming strategies #1206. We will use such information to improve EvoMaster. Thanks!
Unfortunately, still only 1 testcase was generated even after removing all the constraints about length and patterns. By the way, I have submitted the survey.
Here is the new api doc:
openapi: 3.1.0
info:
title: OpenAPI definition
version: v0
servers:
- url: http://localhost:8000
description: Generated server url
paths:
/api/bind_card_apply:
post:
tags:
- demo-controller
operationId: bindCardApply
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/CommonReqBindCardReq'
required: true
responses:
"200":
description: OK
content:
'*/*':
schema:
$ref: '#/components/schemas/CommonRespBindCardResp'
components:
schemas:
BindCardReq:
required:
- bankCardNo
- bankCardPhoneNumber
- idCardNo
- loanPersonName
type: object
properties:
idCardNo:
type: string
loanPersonName:
type: string
bankCardNo:
type: string
bankCardPhoneNumber:
type: string
CommonReqBindCardReq:
type: object
properties:
appId:
type: string
data:
type: string
requestId:
type: string
timestamp:
type: string
key:
type: string
sign:
type: string
bizData:
$ref: '#/components/schemas/BindCardReq'
BindCardResp:
type: object
properties:
code:
type: string
msg:
type: string
sessionId:
type: string
CommonRespBindCardResp:
type: object
properties:
data:
type: string
key:
type: string
sign:
type: string
bizData:
$ref: '#/components/schemas/BindCardResp'
@ytfrank thanks for answering the survey
removing the constraints on schema would make things worse, as I assume that the API is still validating those constraints internally.
regarding regex handling, there is currently a bias in short strings... that might struggle to reach a 16-char long string (the handling of regex in EM should be improved). I would suggest to put back all the constraints, and try to make sure all length-related constraints are expressed in the pattern, eg, pattern: "^[0-9]{16,19}$" to represent min=16 and max=19
Still only saved 1 test, although many tests were evaluated. Is it expected that only one testcase is generated when the response is similar?
Here is the info: (1) part of SUT log 2025-03-24 17:27:59 [http-nio-8080-exec-10] INFO c.e.demo.controller.DemoController - ------Return resp:{}{"data":"S6geXrvlQqX+XXHCX9PSMVQeO9C8QfJXJFBSS2Ef2RbVKf5mQSH0hgoU7eBdy470","key":"fwfMybMfkb4uOT3eeuS4O3jhDWSyik95mb9hlLkkLDOgNgd7X71lVB0qJWzhCn5bEeKWlm1hu3gBLXM45AX44d3I4msMfYV0Cqb+ZzAExDyZ5L7GR4XzChV3f+xJFskHJUb2quv+Tr9TuDBTzWFJUKK+U634VUNA1Ogkkl+TddmSTBfap3rmcO+e+L42G+avjpl9xCd825inCUG8C5WA9ZTa2RpJRcTom76lNEf9He7gK+0pvtM3JQy7BQYXlTwLDuW5L/Wza/33GzXrBNHvGmgawVX3SutBPPhyekjrIlGlEvjsCLRdbUcHpsl2stXwQJan2iI4GK50Eb2HZyPC/Q==","sign":"aXZ3p7DAxubnVfC7ernXA+QBK/JcqX8a8FLBDgfPA17eis6jHIoQdSR3ThWr4gUDJRMv6exCRVdwNU9WVhFkZC4lPDeu9HMrDZrzDZatr0q8tHnK+t6PpF8LUZpJ3MGbZ5ViyL21wqcqtxlndz0emYUoGD+TEykNH7y8roKyJ1PdfWWkrjEVBdlOv5RnSX1OZV/QPjh7a12z+vvF+pFQlugawcMePsMpfi4kBWchfTAXbZjCxSOwaMEKzTg9F6n1ydwKwKAWBdYfNjtsE8J4kw2c7ifRwNCvq5V9ZzlUOdGYBpO++ZhaBdWsKVCqbzZ/rZVYcQeUL781MKaX6Pr2dA=="} 2025-03-24 17:27:59 [qtp1089504328-19] INFO ks.long5.DemoEmbeddedSutController - Resetting state of SUT... 2025-03-24 17:27:59 [http-nio-8080-exec-1] INFO c.e.demo.controller.DemoController - ------Received req:{}{"appId":"KqV","bizData":{"bank_card_no":"682697863096554076","bank_card_phone_number":"17001810112","id_card_no":"239582327441924811","loan_person_name":"130"},"data":"IVxqEl","key":"4c","requestId":"b7QJg","sign":"EM_2391_XYZ","timestamp":"qBkndYxXDOF"} 2025-03-24 17:27:59 [http-nio-8080-exec-1] INFO c.e.demo.controller.DemoController - ------Return resp:{}{"data":"W85yJaLI2Vb3rVZIAK/OM1eVmpsBEiqGSuc3HcIhVgs51pMQpN2occ+ahelT0TMV","key":"dojeVzx8OGNF4TcSXtJqSc1Cm6y35K+GR39yWXFG9Pv0nT47O+KOb+antpwWz6e6/e5oylGpO52FA65DTxogg2XcIvVeR/yH9TzllqMZ6eLyiwme99koM51pmsgV3Nd/RguMNbyV99voYcFbYO/qoT/bvnpOxJIPHCbsgnVVcmoPQWaflxMdrz8ytCRShqqWhKm4iOhtSjomp3AhQmdn9JRI31D7Qqf93ZVrFfakkEgiHfJ1udsSkRjvCw/ZV8fJei0SzujZqVZG6LWycEtDgR252mrNj/RyPg1lab/SazLWfkcgFuIaldst3s4D2pfhCE4KfL3QkrYzm76pwMvS/Q==","sign":"P6W3nI8DnUAgeSO8QC00Xki6wyxJhPuEZlVV9pFRXUz/cWykrycu0+YjuxtLet/KrjwRvwtb0za8wBxvNHhc8xs+2XRsW+xv1qNfzTuyWfAytY5mnZCsh278ndM2+lEVkyVYw3HikMv/Cp/nWoEekRhC509qINMITjoYyIrdlGH2MN5uuQ1aYYhlL5sB7GL7lxLK7jJoAvBcmd7kF4sYbZ90+g8WBAp47V4J+S5nCFM9A19ZQCYranZej+qqvGAy6djYj3VpZTEOPxXCj+3kuThJbozbZQudzVT4BGts6YZsjl3wktfXRQYtICIwHjbWKVtRzLzwZZsoKzYFR52YhA=="} 2025-03-24 17:27:59 [qtp1089504328-18] INFO ks.long5.DemoEmbeddedSutController - Resetting state of SUT... 2025-03-24 17:27:59 [http-nio-8080-exec-2] INFO c.e.demo.controller.DemoController - ------Received req:{}{"appId":"EM_2390_XYZ","bizData":{"bank_card_no":"682697863096554076","bank_card_phone_number":"17001810112","id_card_no":"239582327441924811","loan_person_name":"130"},"data":"JVxqEl","key":"4d","requestId":"b7QJg","sign":"EM_2391_XYZ","timestamp":"qBkndYxXDOF"}
2025-03-24 17:27:59 [http-nio-8080-exec-2] INFO c.e.demo.controller.DemoController - ------Return resp:{}{"data":"Udq+Pxj63RniizY9hJ+jiHmkUbo5fxygW9JTWuLdwPGOlMX3lflxXNoKg9O8DiPl","key":"bJAhDO2ZBtIW1oe0vBvMCV2BESMqnejfXBhGEkiSylpkLq3NhWXFZRIlWJAY2i+Na9yTBQRSReGpXPZP48VdxF9M58Te5OD6ep0rlfKADitj7IEJh/oTkGRX+08QfDmcdDnbr2po4odJUW7NRxn3rckuOWM42GFLb5GXSNpyoVc4laOJRASuM1ObxdyPQF1cTgp7Xn6kEzWWKv2a9PU8RZ13ynTOJ/AEPYKdK3Z6itEM2SJtRFeQKlcH5IFSwFFZ19c4BhRNbVWIJvAXundvsWQR6HN3PwEGGIQkbOJuWJK9q+jeJ15FGuRBSNgdCunSwHi/w1KE26nQNEG9kp6Szw==","sign":"BP4OHeOEHshegDft+JFp9PMG+Z1hXqYXT8bUHks1oa3zHUiOWrFxojERf+/Fdmtvq0/K5YvaQ+B3D6rWShyKuDJfU0ltNHebHmrFyZB5nnGzEI4kZOXYZnykVxroh9/4KbXJT4nuvpaH07qo0fIaC0WPFXfJr8cFVN5jiZnhqS8A8qrkfBE3ezo0EazKz51HTL3xE9jzQSH+kV1oquig/hUazL7alGJODLcn+u5n2sJYqxsRJoeczbDLBBV3lMne7bzs+cAjOmsABcbVoXmAm8IhRV1mbqVMBqcWrZB1XXTSYm8YFsIys+vRQRN7Eoh6GdSEIixufCcb5bsCPO1t2w=="} 2025-03-24 17:27:59 [qtp1089504328-19] INFO ks.long5.DemoEmbeddedSutController - Resetting state of SUT... 2025-03-24 17:27:59 [http-nio-8080-exec-3] INFO c.e.demo.controller.DemoController - ------Received req:{}{"appId":"EM_2390_XYZ","bizData":{"bank_card_no":"682697863096554076","bank_card_phone_number":"17001810112","id_card_no":"239582327441924811","loan_person_name":"130"},"data":"IVxqEl","key":"4c","requestId":"b7QJg","sign":"EM_2391_XYZ","timestamp":"qBkndYxXDOFM"}
(2) openapi doc openapi: 3.1.0 info: title: OpenAPI definition version: v0 servers:
- url: http://localhost:8000 description: Generated server url paths: /api/bind_card_apply: post: tags: - demo-controller operationId: bindCardApply requestBody: content: application/json: schema: $ref: '#/components/schemas/CommonReqBindCardReq' required: true responses: "200": description: OK content: '/': schema: $ref: '#/components/schemas/CommonRespBindCardResp' components: schemas: BindCardReq: required: - bankCardNo - bankCardPhoneNumber - idCardNo - loanPersonName type: object properties: idCardNo: pattern: "^[0-9]{17}[0-9Xx]$" type: string loanPersonName: pattern: "^[0-9]{1,20}$" type: string bankCardNo: pattern: "^[0-9]{16,19}$" type: string bankCardPhoneNumber: pattern: "^[1][3-9][0-9]{9}$" type: string CommonReqBindCardReq: type: object properties: appId: type: string data: type: string requestId: type: string timestamp: type: string key: type: string sign: type: string bizData: $ref: '#/components/schemas/BindCardReq' BindCardResp: type: object properties: code: type: string msg: type: string sessionId: type: string CommonRespBindCardResp: type: object properties: data: type: string key: type: string sign: type: string bizData: $ref: '#/components/schemas/BindCardResp'
(3) the saved testcase @Test(timeout = 60000) public void test_0() throws Exception {
given().accept("*/*")
.header("x-EMextraHeader123", "")
.contentType("application/json")
.body(" { " +
" \"appId\": \"_EM_0_XYZ_\", " +
" \"data\": \"j66xS1uTWaMNar01\", " +
" \"requestId\": \"_EM_1_XYZ_\", " +
" \"timestamp\": \"_EM_2_XYZ_\", " +
" \"key\": \"Kon_BuCkGdO8a\", " +
" \"sign\": \"_EM_3_XYZ_\", " +
" \"bizData\": { " +
" \"idCardNo\": \"810842383106002679\", " +
" \"loanPersonName\": \"5\", " +
" \"bankCardNo\": \"9572223310436983\", " +
" \"bankCardPhoneNumber\": \"13894612629\" " +
" } " +
" } ")
.post(baseUrlOfSut + "/api/bind_card_apply?EMextraParam123=_EM_4_XYZ_")
.then()
.statusCode(200)
.assertThat()
.contentType("application/json")
.body("'data'", containsString("eYy2Vo8ShMKBXBW90MnEWULcAXAYHV5hGPs37+eRgHRmqe8Ma6OafcCpt0dQqQjQ"))
.body("'key'", containsString("TgKXdcHxXjW5pTgx8w8aUK24GZJFrx4UkK3WWDav/FDh7/c3rR8aB1xTbQRgKAHI9zszaE0IY+5XxjyPVT8pa9xr4xJKLN8U2/wyDv4FUA8lemwUymEtK5DaBecGww+YI+izTQl9Wu2/e5TE2J5LJTpRu/ENzBkwqCnZTv4CfuAN6cISMouOXmL0+mlCx6qRpEUz1jEXWgv/Nki4nReParUG/6SPQgIEudonqHA4m8t4rVbigoSsp7g/E8dkuyFN6pFw/fPerGE1pzJrUUHZcI5h9qpI7HUJYbNQ6+F42KgRqJBoZqPN+4CvPDNG6pADIEiE1cZK4NS9DG5MXOqcYw=="))
.body("'sign'", containsString("T6bhqZArJ4X8/nZa/ETNDL352J28lf/Dxbdewc2k57RV2QFK4o/WsEdNHx0SA0wg4ejZzZpTEGeaZvmDT9QEM/dIGM4/5xlgbXyIPtOogiLXz9cUZ3zLaP7ygBajikvVMH7hRVOh51PQPHAWbSqO8uSkvzFEbHxjI9tMC55D3agYBzCyEjDIS77GhKwX1j7xF6LhFEfv5IG2DqSXr5Rsaj/uydXH1iCAhDgqO2aWXO1vnBmPmkwP5hHG+bUPbYTxn0OCZz058ZaeyTvsAkRDm1xoK57/Sz0gu3h/vyb4OpfpbcMird+BDjrTRFx5hphzZVUd/xxP0JcU0BgKOYcIMw=="))
.body("'bizData'", nullValue());
}
hi,
it is weird. I would need to look at the source code of the API to be able to tell what is going on.
You could refer the source code in EvoMaster/e2e-tests/spring-rest-rsa/src/main/java/com/example/demo. I just ran the testing against it, and also 1 test was saved.
- Starting to generate test cases
- Consumed search budget: 99.463%
- Covered targets: 13; time per test: 6.3ms (1.0 actions); since last improvement: 88s
- Starting to apply minimization phase
- Recomputing full coverage for 1 tests
- Recomputing coverage did lose many targets, more than the threshold 20.0%: from 13 to 7, i.e., lost 6
- Missing targets:
- Line_at_com.example.demo.controller.DemoController_00031
- Success_Call_at_com.example.demo.controller.DemoController_00031_0
- Line_at_com.example.demo.controller.DemoController_00032
- Line_at_com.example.demo.controller.DemoController_00033
- Success_Call_at_com.example.demo.controller.DemoController_00033_0
- Success_Call_at_com.example.demo.controller.DemoController_00033_1
- No test to minimize
- Minimization phase took 0 seconds
- Evaluated tests: 6549
- Evaluated actions: 6950
- Needed budget: 0%
- Passed time (seconds): 90
- Execution time per test (ms): Avg=10.43 , min=4.00 , max=1333.00
- Execution time per action (ms): Avg=9.77 , min=4.00 , max=1333.00
- Computation overhead between tests (ms): Avg=3.29 , min=1.00 , max=56.00
- Computation overhead of resetting the SUT (ms): Avg=1.76 , min=0.00 , max=202.00
- Computation overhead of fetching test results, per test, subset of targets (ms): Avg=1.21 , min=0.00 , max=29.00
- Going to save 1 test to /Users/apple/gitrepo/EvoMaster/whitebox-deom/src/main/java/ks/long9
- Potential faults: 2
- Covered targets (lines, branches, faults, etc.): 10
- Bytecode line coverage: 2% (1 out of 41 in 1 units/classes)
- Successfully executed (HTTP code 2xx) 0 endpoints out of 1 (0%)
2 tests were saved after adding a valid signature in the openapi doc: sign: type: string
-
enum: -
- +"Svlk4DIoAO0Q2cAU5ZLS0uESjKI4fz9TE7yzyKAZSRlVyJWMyaX7ZxzqUMAWusWV7+W8jLk2lNgP+Fzg8yf/OzKpz8R45ih94sBanvhClOdL6m9J8iY2+hxINaHme9YIdGnHAU5Sg47BAqlljrvKMEWcEsEDa25OMVzMeHPCig24PjurnfGHWmwEGJ/p9TZvcAOQPkmaAeRK2IfH/3aNzJNYJDNOGtETxGFbg0A54ry8+GGtdPtiY8yr5JLkVT6nGaRpZN2VQBgcbE9Uxlrx80RiHRbtR87i8rT95kKv1OnrlxTG/U+j1smuBNDqWlH65n829LvO2yIefM7RKiLt+Q=="
It seems only one testcase is generated when the coverage is identical and the response is similar. I think it's OK and we just need to cover more targets.