Support for generating inputs dealing with RSA encryption.

[ytfrank]

A classic scenario is that the other party first encrypts the data, signs all the business parameters with its own RSA private key, and then puts the value of this signature into the parameter "sign". Therefore, after receiving the request in the API, it will first verify whether the "sign" in the input parameters is correct and then decrypt it. If any step is wrong, API will return and the later codes can not be covered.

One example of verifying the sign: // java.security.Signature public static boolean verify(byte[] data, byte[] sign, PublicKey publicKey) throws Exception { Signature signature = Signature.getInstance("SHA256withRSA"); signature.initVerify(publicKey); signature.update(data); return signature.verify(sign); }

Other important info:

• version of EvoMaster (EM) used : 3.3.0
• how EM is run (eg, if from JAR or from one of its OS installers) : white-box
• version of applicable runtimes (eg, JVM, NodeJS and .Net). For Java, can paste the output of java --version : 1.8.0
• command-line options used to run EM

[arcuri82]

hi @ytfrank , currently there is no way in EM to do something like this. it seems like a very special scenario. Haven't heard of doing something similar before in other APIs... or maybe it is something common in a specific domain / business? Automatically reversing/inferring RSA private keys is simply infeasible. Such info would have to be provided to EM, somehow. It could be done, but, to be honest, it is not a feature that we would prioritize, unless we see more APIs that require such a feature.
Such feature could be implemented as part of a so-called "academia-industry" collaboration (see here for details). If that is something that could be of interest, you can contact me at [email protected] (as GitHub does not have private message support)

[ytfrank]

Thanks for your info. In the field of financial credit, data encryption and signature before transmission are common practices. I'd like to study how to provide the info to EM first, and then evaluate the proper way.

[arcuri82]

hi @ytfrank are there any example / proof-of-concept open-source APIs with these properties that I can look at? or is there any closed-source, industrial API that has its OpenAPI schema online that I can look at? with a concrete example, I could try to estimate how much work would be needed to add such a feature

[ytfrank]

Here is an OpenAPI example:

openapi: 3.0.1
info:
  title: OpenAPI definition
  version: v0
servers:
  - url: http://localhost:8080
paths:
  /api/bind_card_apply:
    post:
      operationId: bindCard_1
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Req'
        required: true
      responses:
        "200":
          description: OK
          content:
            '*/*':
              schema:
                $ref: '#/components/schemas/Resp'
components:
  schemas:
    Req:
      type: object
      properties:
        appId:
          type: string
          description: Application ID
        data:
          type: string
          description: Encrypt data using AES symmetric encryption
        requestId:
          type: string
          description: Request ID
        timestamp:
          type: string
          description: Timestamp
        key:
          type: string
          description: The encrypted AES key， encrypted by the other party's RSA public key
        sign:
          type: string
          description: Signature of the data, used to verify the integrity of the request
        bizData:
          $ref: '#/components/schemas/BindCardReq'
          description: Plaintext data, used to store the request data after decryption
    BindCardReq:
      type: object
      properties:
        loanPersonName:
          type: string
          description: Borrower's name
          maxLength: 30
          minLength: 3
        bankCardNo:
          type: string
          description: Bank card number, used to bind the bank card
          maxLength: 19
          minLength: 16
          pattern: "^[0-9]+$" # Supports digits only
        bankCardPhoneNumber:
          type: string
          description: Bank card reserved phone number, used to receive verification codes
          maxLength: 12
          minLength: 10
    Resp:
      type: object
      properties:
        data:
          type: string
          description: Encrypt data using AES symmetric encryption
        key:
          type: string
          description: The encrypted AES key， encrypted by the other party's RSA public key
        sign:
          type: string
          description: Signature of the data, used to verify the integrity of the response
        bizData:
          $ref: '#/components/schemas/BindCardResp'
    BindCardResp:
      type: object
      properties:
        code:
          type: string
          description: Response status code, indicating the result of the request processing
        msg:
          type: string
          description: Response message, describing the details of the request processing
        sessionId:
          type: string
          description: session ID, used to track the verification code request
          maxLength: 64

---

Encryption Process The encryption process consists of the following steps:

1. Generate an AES KEY.

2. Encrypt the concatenated business parameters using the AES Key provided by the merchant to obtain a byte array.

AES(json parameters)

3. Encode the byte array using Base64 to get a string, and use this string as the Params field.

Base64(AES(json parameters))

4. Use the merchant's public key to encrypt the AES Key with RSA and then encode it with Base64. The encrypted AES Key will be used as the key field.

Base64(RSA(AES KEY))

After completing the above steps, we obtain the encrypted business parameters and secret key. Finally, these encrypted business parameter values are added to the system parameters.

Signature Process
Note: Signing is performed on the encrypted data.

1. First, sort all parameters except sign in ascending order by letter, and then concatenate them using &.

appId=123&key=a&params=encryptData{"a":"b"}&timestamp=1&version=1.0

2. Use the SHA1WithRSA algorithm and your private key to sign the concatenated string, resulting in a byte array.

SHA1WithRSA(appId=123&key=a&params=encryptData{"a":"b"}&timestamp=1&version=1.0)

3. Encode the byte array using Base64URLSafe to obtain a signature string.

Base64URLSafe(SHA1WithRSA(appId=123&key=a&params=encryptData{"a":"b"}&timestamp=1&version=1.0))

After completing the above three steps, we obtain the signature of the business parameters. Finally, the signature value is added to the system parameter sign.

sign=Base64URLSafe(SHA1WithRSA(appId=123&key=a&params=encryptData{"a":"b"}&timestamp=1&version=1.0))

I hope there will be a mechanism to inform EM how to prepare specific data during initialization, and the specific implementation is coded by the user.

[arcuri82]

hi @ytfrank ,

thanks for the explanation. Adding a feature to support this would be possible. In the driver, we could add a method to specify if any parameter is "derived" from others, and an abstract method to "infer" a derived field from the values of test case (eg JSON string representation of the object), for the user to implement. Then, each time we need to specify/mutate the value of a "derived" field, we call the "infer" method to create the actual value.

So yes, it is technically possible to add such feature. But, without APIs to use for experiments requiring such feature (e.g., open-source on GitHub, or in academia-industry collaborations), in all honesty it would be hard to prioritize this feature compared to other pressing ones :(

[ytfrank]

Thanks for your reply. It's somewhat difficult to access the company's code, so a demo open-source project has been prepared for testing: https://github.com/ytfrank/RestDemo

The openapi doc can be seen from http://localhost:8080/v3/api-docs after the app starts.

openapi: 3.0.1 info: title: OpenAPI definition version: v0 servers:

• url: http://localhost:8080 description: Generated server url paths: /api/bind_card_apply: post: tags: - demo-controller operationId: bindCardApply requestBody: content: application/json: schema: $ref: '#/components/schemas/CommonReqBindCardReq' required: true responses: "200": description: OK content: '/': schema: $ref: '#/components/schemas/CommonRespBindCardResp' components: schemas: BindCardReq: required: - bankCardNo - bankCardPhoneNumber - idCardNo - loanPersonName type: object properties: idCardNo: maxLength: 18 minLength: 18 pattern: "^[0-9]{17}[0-9Xx]$" type: string loanPersonName: maxLength: 20 minLength: 0 type: string bankCardNo: maxLength: 19 minLength: 16 pattern: "^[0-9]+$" type: string bankCardPhoneNumber: maxLength: 11 minLength: 11 pattern: "^[1][3-9][0-9]{9}$" type: string CommonReqBindCardReq: type: object properties: appId: type: string data: type: string requestId: type: string timestamp: type: string key: type: string sign: type: string bizData: $ref: '#/components/schemas/BindCardReq' BindCardResp: type: object properties: code: type: string msg: type: string sessionId: type: string CommonRespBindCardResp: type: object properties: data: type: string key: type: string sign: type: string bizData: $ref: '#/components/schemas/BindCardResp'

[arcuri82]

@ytfrank many thanks for providing a working example. I ll have a look at it. although this month is quite busy... might try to support this sometime during February. meanwhile, in that repository, could you specify the license of that code? (eg, if open-source license like Apache or LGPL) I might integrate parts of it inside EvoMaster for its own E2E tests to check this new functionality support (eg, look under e2e-tests folder, where we have hundreds of artificial APIs to test EvoMaster). alternative, if you want/prefer, you could make a PR of that API example into EvoMaster (eg, as new module under e2e-tests )

[arcuri82]

once we support this in EvoMaster, we might also consider to add it to https://github.com/WebFuzzing/EMB for experimentation

[ytfrank]

I just made a PR: https://github.com/WebFuzzing/EvoMaster/pull/1158

[arcuri82]

@ytfrank thx! hopefully i ll manage to get sometime to support this by the end of February

[ytfrank]

You're welcome! Hope you can find time to support this. Let me know if you need anything.

[ytfrank]

Hi! Any progress so far?

[arcuri82]

Hi @ytfrank , unfortunately, there has been no progress :( We are stuck on some other high-priority tasks that are taking much longer than expected. Based on those, might manage to get it done during April, and hopefully no later than May

[arcuri82]

hi @ytfrank ,

I finally have sometime available to look into this feature request.

I have a question though. In the example you created, is bizData actually sent with the request? or is it expected to be left empty?

I mean, if it is sent with the request, wouldn't the controller (i.e., DemoController.bindCardApply) then need to check if the content of the decrypted data match what is inside bizData?

If it is not expected to be part of the request (ie its field bizData is left null, and only encrypted data is populated), would then the controller need to unmarshall the received decrypted data into a BindCardReq object using a JSON library?

[ytfrank]

I'm really glad to hear that you'll be able to review it.

Your guess is correct. bizData is not part of the request. The controller will put the decrypted data into bizData, and this operation is omitted in the demo code.

[arcuri82]

hi @ytfrank ,

now in master branch we have support to handle this. to try it out, you ll need to build SNAPSHOT locally, as might take sometime before we make a new release. You can check e2e-tests/spring-rest-rsa/src/test/java/com/example/demo/controller/EmController.java, in particular there are 2 things to do:

1. declare which parameters need to be derived, and their order, which is done in getProblemInfo()
2. specify how parameters are derived (done by overriding deriveObjectParameterData())

the test e2e-tests/spring-rest-rsa/src/test/java/org/evomaster/e2etests/spring/rest/rsa/RsaEMTest.java shows that we can handle the example you created.

Two related notes:

1. you REST controller was always returning 200... so made sure returning 400 in case of user error (make easier to check what is covered)
2. the use of UUID.randomUUID() in the responses is problematic, as it makes the assertion flaky. we are aware of the issue, but will take sometime before we can address it properly

Let me know if this fixes your issues in using EvoMaster. thx

[ytfrank]

Great! The issue has been resolved. Thank you very much!

[arcuri82]

closed as release 4.0.0 is out