fcli icon indicating copy to clipboard operation
fcli copied to clipboard

fcli error when uploading ZIP file with 18MB, but working if it is around 2MB

Open jechtslasher85 opened this issue 7 months ago • 14 comments

Current Behavior

Scenarios Taken: SAST scan via powershell command (using fcli v3.5) with 18MB zip file = Failed SAST scan via powershell command with 2MB zip file = Success SAST scan via GitLab CICD Pipeline, using fcli (fcli v3.4) commands, with 18MB zip file = Failed SAST scan via GitLab CICD Pipeline, using fcli commands, with 2MB zip file = Success SAST scan via Fortify-On-Demand UI (Website), uploaded the 18MB zip file = Success I think there is no issue with the zip file since it got scanned via the FOD SAST website. For these tests, I have used two email accounts since there is a limit on the Assessment Units per trial account

The error is this:

Upload feedback-form-backend_05-26-2025.zip: 0 of 19043360 bytes complete
Upload feedback-form-backend_05-26-2025.zip: 1048576 of 19043360 bytes complete
FcliSimpleException: Error uploading file
	at com.fortify.cli.fod._common.rest.helper.FoDFileTransferHelper.uploadChunked(FoDFileTransferHelper.java:101)
Caused by: com.fortify.cli.common.rest.unirest.UnexpectedHttpResponseException:   
  Request: POST https://api.trial.fortify.com/api/v3/releases/246299/static-scans/start-scan-with-defaults?isRemediationScan=false&scanTool=fcli&scanToolVersion=3.4.1&scanMethodType=Other&fragNo=2&offset=2097152: 
  Reason: HTTP 500 Internal Server Error
  Body: 
    {"errors":[{"errorCode":2027,"message":"Stream out of sync"}]}
  ----
	at com.fortify.cli.common.rest.unirest.config.UnirestUnexpectedHttpResponseConfigurer$UnexpectedHttpResponseInterceptor.onResponse(UnirestUnexpectedHttpResponseConfigurer.java:[36](https://gitlab.com/wph_gitlab/hdb-project/feedback-form-backend-test/-/jobs/10150352182#L36))

Expected Behavior

File upload should be successful since the limit for trial account is 150MB file size. I also have at least 2 available Assessment Units so there is no reason for it to fail.

Steps To Reproduce

SAST scan via powershell command (using fcli v3.5) with 18MB zip file = Failed SAST scan via powershell command with 2MB zip file = Success SAST scan via GitLab CICD Pipeline, using fcli (fcli v3.4) commands, with 18MB zip file = Failed SAST scan via GitLab CICD Pipeline, using fcli commands, with 2MB zip file = Success SAST scan via Fortify-On-Demand UI (Website), uploaded the 18MB zip file = Success

Environment

Did this on the following env:
1. Laptop with Windows 11 OS, using fcli commands via powershell, fcli version is v3.5
2. GitLab CICD Pipeline, stage is using the image:fortifydocker/fortify-ci-tools:latest, using  fcli commands, fcli version is v3.4

Anything else?

I have raised this to openttext team and they told me to raise this here as well

jechtslasher85 avatar May 29 '25 01:05 jechtslasher85

@jechtslasher85 Thanks for reporting this. We'll investigate and keep you updated.

rsenden avatar May 29 '25 09:05 rsenden

@jechtslasher85 If you have a chance, can you please try again? The FoD team has adjusted some networking settings on the trial instance, which should prevent these 'stream out of sync' errors.

rsenden avatar Jun 03 '25 08:06 rsenden

hi @rsenden , i have tried it again and it is now properly uploading the files. thank you

jechtslasher85 avatar Jun 05 '25 05:06 jechtslasher85

hi @rsenden, I was having the same problem as jechtslasher85, also stopping exactly at 1048576 bytes, but since that problem has been resolved, it is now always stopping at 41943040 bytes

   Upload package.zip: 41943040 of 43690890 bytes complete
  java.lang.RuntimeException: Error uploading file
  	at com.fortify.cli.fod._common.rest.helper.FoDFileTransferHelper.uploadChunked(FoDFileTransferHelper.java:100)
  	at com.fortify.cli.fod._common.scan.helper.sast.FoDScanSastHelper.startScan(FoDScanSastHelper.java:83)
  	at com.fortify.cli.fod._common.scan.helper.sast.FoDScanSastHelper.startScanWithDefaults(FoDScanSastHelper.java:55)
  	at com.fortify.cli.fod.sast_scan.cli.cmd.FoDSastScanStartCommand.startScan(FoDSastScanStartCommand.java:71)
  	at com.fortify.cli.fod._common.scan.cli.cmd.AbstractFoDScanStartCommand.getJsonNode(AbstractFoDScanStartCommand.java:36)
  	at com.fortify.cli.fod._common.output.cli.cmd.AbstractFoDJsonNodeOutputCommand.getJsonNode(AbstractFoDJsonNodeOutputCommand.java:23)
  	at com.fortify.cli.common.output.cli.cmd.AbstractOutputCommand.call(AbstractOutputCommand.java:33)
  	at com.fortify.cli.common.output.cli.cmd.AbstractOutputCommand.call(AbstractOutputCommand.java:22)
  	at picocli.CommandLine.executeUserObject(CommandLine.java:2118)
  	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2538)
  	at picocli.CommandLine$RunLast.handle(CommandLine.java:2530)
  	at picocli.CommandLine$RunLast.handle(CommandLine.java:2492)
  	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2350)
  	at picocli.CommandLine$RunLast.execute(CommandLine.java:2494)
  	at picocli.CommandLine.execute(CommandLine.java:2247)
  	at com.fortify.cli.app.runner.DefaultFortifyCLIRunner.run(DefaultFortifyCLIRunner.java:59)
  	at com.fortify.cli.app.FortifyCLI.execute(FortifyCLI.java:38)
  	at com.fortify.cli.app.FortifyCLI.main(FortifyCLI.java:32)
  	at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)
  Caused by: com.fortify.cli.common.rest.unirest.UnexpectedHttpResponseException: 
  Request: POST https://api.trial.fortify.com/api/v3/releases/246166/static-scans/start-scan-with-defaults?isRemediationScan=false&scanTool=fcli&scanToolVersion=2.12.2&scanMethodType=Other&fragNo=-1&offset=42991616: 
  Response: 500 Internal Server Error
  Response Body:
  {"errors":[{"errorCode":1001,"message":"Unexpected error processing request"}]}
  	at com.fortify.cli.common.rest.unirest.config.UnirestUnexpectedHttpResponseConfigurer$UnexpectedHttpResponseInterceptor.onResponse(UnirestUnexpectedHttpResponseConfigurer.java:36)
  	at kong.unirest.CompoundInterceptor.lambda$onResponse$1(CompoundInterceptor.java:48)
  	at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1596)
  	at kong.unirest.CompoundInterceptor.onResponse(CompoundInterceptor.java:48)
  	at kong.unirest.apache.ApacheClient.request(ApacheClient.java:134)
  	at kong.unirest.Client.request(Client.java:57)
  	at kong.unirest.BaseRequest.request(BaseRequest.java:365)
  	at kong.unirest.BaseRequest.asString(BaseRequest.java:218)
  	at com.fortify.cli.fod._common.rest.helper.FoDFileTransferHelper.uploadChunked(FoDFileTransferHelper.java:93)
  	... 18 more '''

Im using a free trial of Fortify on Demand, and I'm trying to implement a CICD using the Fcli github Actions

RSmistia avatar Jun 05 '25 08:06 RSmistia

@RSmistia Thanks for reporting. The error message is different, so there might be a different underlying reason. Can you consistently reproduce this issue, or was this a one-time error?

While waiting for you response, I'll also contact the FoD team again to get their input.

rsenden avatar Jun 05 '25 09:06 rsenden

@rsenden I have tried multiple times, and every time it fails with that exact bytes sent. Thank you in advance!

RSmistia avatar Jun 05 '25 11:06 RSmistia

@RSmistia I'll ping the FoD team once again to see whether they have any updates on this issue. In the meantime, any updates from your side? Can you try once more and let us know whether the issue still exists?

rsenden avatar Jun 16 '25 09:06 rsenden

@rsenden Sorry for the late reply, it seems the problem has been fixed, and I haven't changed anything, I can now do the scan correctly without any errors, thank you for all the help!

RSmistia avatar Jun 18 '25 09:06 RSmistia

@rsenden Retracting my previous statement, the error persists, don't know why it was working that day, but it is not working again.

RSmistia avatar Jun 20 '25 15:06 RSmistia

@RSmistia I'll ping the FoD team once again to see whether they have any updates on this issue. In the meantime, any updates from your side? Can you try once more and let us know whether the issue still exists?

Hi @rsenden, I am experiencing the same error when uploading a large ZIP file with fcli (v3.6.0). Log excerpt:

Upload package.zip: 130023424 of 131681596 bytes complete
  FcliSimpleException: Error uploading file
  	at com.fortify.cli.fod._common.rest.helper.FoDFileTransferHelper.uploadChunked(FoDFileTransferHelper.java:101)
  Caused by: com.fortify.cli.common.rest.unirest.UnexpectedHttpResponseException:   
  Request: POST https://api.ams.fortify.com/api/v3/releases/***/static-scans/start-scan-with-defaults?isRemediationScan=false&scanTool=fcli&scanToolVersion=3.6.0&scanMethodType=Other&notes=Triggered+by+GitHub+Actions+(https://github.com/<>)&fragNo=-1&offset=131072000: 
RUN POLICY_CHECK: /Users/runner/work/_temp/fortify/tools/fcli/****/bin/fcli fod action run check-policy --rel ***
    Reason: HTTP 500 Internal Server Error
    Body: 
      {"errors":[{"errorCode":1001,"message":"Unexpected error processing request"}]}
    ----
  	at com.fortify.cli.common.rest.unirest.config.UnirestUnexpectedHttpResponseConfigurer$UnexpectedHttpResponseInterceptor.onResponse(UnirestUnexpectedHttpResponseConfigurer.java:36)
  	at kong.unirest.CompoundInterceptor.lambda$onResponse$1(CompoundInterceptor.java:48)
  	at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1596)
  	at kong.unirest.CompoundInterceptor.onResponse(CompoundInterceptor.java:48)
  	at kong.unirest.apache.ApacheClient.request(ApacheClient.java:134)
  	at kong.unirest.Client.request(Client.java:57)
  	at kong.unirest.BaseRequest.request(BaseRequest.java:365)
  	at kong.unirest.BaseRequest.asString(BaseRequest.java:218)
  	at com.fortify.cli.fod._common.rest.helper.FoDFileTransferHelper.uploadChunked(FoDFileTransferHelper.java:94)
  	at com.fortify.cli.fod._common.scan.helper.sast.FoDScanSastHelper.startScan(FoDScanSastHelper.java:84)
  	at com.fortify.cli.fod._common.scan.helper.sast.FoDScanSastHelper.startScanWithDefaults(FoDScanSastHelper.java:56)
  	at com.fortify.cli.fod.sast_scan.cli.cmd.FoDSastScanStartCommand.startScan(FoDSastScanStartCommand.java:72)
  	at com.fortify.cli.fod._common.scan.cli.cmd.AbstractFoDScanStartCommand.getJsonNode(AbstractFoDScanStartCommand.java:40)
  	at com.fortify.cli.fod._common.output.cli.cmd.AbstractFoDJsonNodeOutputCommand.getJsonNode(AbstractFoDJsonNodeOutputCommand.java:23)
  	at com.fortify.cli.common.output.cli.cmd.AbstractOutputCommand.call(AbstractOutputCommand.java:34)
  	at com.fortify.cli.common.output.cli.cmd.AbstractOutputCommand.call(AbstractOutputCommand.java:23)
  	at picocli.CommandLine.executeUserObject(CommandLine.java:2118)
  	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2538)
  	at picocli.CommandLine$RunLast.handle(CommandLine.java:2530)
  	at picocli.CommandLine$RunLast.handle(CommandLine.java:2492)
  	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2350)
  	at picocli.CommandLine$RunLast.execute(CommandLine.java:2494)
  	at picocli.CommandLine.execute(CommandLine.java:2247)
  	at com.fortify.cli.app.runner.DefaultFortifyCLIRunner.run(DefaultFortifyCLIRunner.java:63)
  	at com.fortify.cli.app.FortifyCLI.execute(FortifyCLI.java:38)
  	at com.fortify.cli.app.FortifyCLI.main(FortifyCLI.java:32)
  Loading action check-policy
  	at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)

The error always happens near the end of the upload, and the file size is approx. 125 MB.

AlejandroLemusR avatar Oct 09 '25 14:10 AlejandroLemusR

@AlejandroLemusR Looks like some of the output is intermingled, showing both the upload failure and start of POLICY_CHECK. Any chance you can reproduce this with 'clean' error output, for example by just manually running the upload command?

Given that this seems to be an issue on the FoD side, and (contrary to the previous reports) you are not using a trial FoD instance, can you please also open an FoD ticket, and include a link to this GitHub issue and any information that might be useful for the FoD team for investigation (ams.fortify.com instance, your tenant, one or more release id's on which this issue shows up, ...)

rsenden avatar Oct 09 '25 15:10 rsenden

Thanks! @rsenden I’ll work on this during the day and will also open the FoD ticket as suggested. I was just waiting for GitHub to be fully back online.

AlejandroLemusR avatar Oct 09 '25 15:10 AlejandroLemusR

@rsenden I have replicated the solution suggested in fortify/fcli#507.
Below are the steps I followed in the GitHub Actions workflow:

- name: 🧰 Setup Java
  uses: actions/setup-java@v5
  with:
    distribution: 'temurin'
    java-version: '21'

- name: Setup Fortify tools
  run: |
    curl -LO https://github.com/fortify/fcli/releases/download/v3.9.1/fcli-mac.tgz
    tar xzvf fcli-mac.tgz
    mv fcli /usr/local/bin
    chmod +x /usr/local/bin/fcli

    curl -LO https://tools.fortify.com/scancentral/Fortify_ScanCentral_Client_Latest_x64.zip
    unzip Fortify_ScanCentral_Client_Latest_x64.zip -d scancentral
    chmod a+x scancentral/bin/scancentral

- name: Download FoDUpload (Java .jar)
  run: |
    curl -LO https://github.com/fod-dev/fod-uploader-java/releases/latest/download/FoDUpload.jar

- name: Login to Fortify
  run: |
    fcli fod session login -u "${{ secrets.FOD_USER }}" -p "${{ secrets.FOD_PAT }}" -t "${{ secrets.FOD_TENANT }}" --url "https://api.ams.fortify.com"

- name: Run scancentral
  run: |
    scancentral/bin/scancentral package -o package.zip -bt dotnet -bf <file>.csproj

- name: Perform SAST Scan
  run: java -jar FoDUpload.jar -z package.zip -ep 2 -aurl "https://api.ams.fortify.com" -purl "https://ams.fortify.com" -tc "${{ secrets.FOD_TENANT }}" -ac "${{ secrets.FORTIFYONDEMAND_CLIENT_ID }}" "${{ secrets.FORTIFYONDEMAND_CLIENT_SECRET }}" -rid "${{ secrets.FOD_RELEASE }}" -n "Triggered by GitHub Actions (${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})"

- name: Logout from Fortify
  if: always()
  run: |
    fcli fod session logout

Result:
The package was uploaded successfully, as shown in the following log:

Image

The file was uploaded correctly and, in the Fortify web UI, it is currently showing as "being analyzed."

Image

AlejandroLemusR avatar Oct 09 '25 21:10 AlejandroLemusR

@AlejandroLemusR, thanks for the update. In your current workflow, you're no longer using fcli to perform any tasks, so if you'd like to stick to this approach, you can remove all fcli-related steps (download/unpack/login/logout).

Of course, we'd like to get this issue resolved though, as we plan on moving most CI/CD integrations to use fcli under the hoods. So, it would be much appreciated if you can help us gather more information on this issue in order to find the root cause.

FoDUploader uses a different FoD endpoint and different upload implementation/HTTP library, so the fact that uploads don't cause errors with FoDUploader doesn't necessarily mean that this is an fcli bug.

Can you please provide some more information on your previous fcli-based approach, like the options being passed on the fcli fod sast-scan start command?

rsenden avatar Oct 09 '25 22:10 rsenden