toolhive icon indicating copy to clipboard operation
toolhive copied to clipboard

Signing of ToolHive CLI .exe For Windows

Open ChrisJBurns opened this issue 4 months ago • 1 comments

WinGet CLI Package Updates Blocked Due to Windows Defender Detections

Summary

WinGet package updates for the ToolHive CLI are consistently getting stuck in the publishing pipeline. Multiple versions (0.3.6, 0.3.7, 0.3.9, 0.3.11) have failed to publish, with the latest available version on WinGet remaining at 0.3.8.

Background

A WinGet moderator confirmed that the delays are primarily caused by Windows Defender false positive detections, which typically clear after a few days. However, this pattern has now affected four consecutive releases, suggesting a systemic issue rather than random occurrences.

Affected Versions

  • v0.3.6 - PR #299672 (opened Sep 30) - Eventually merged after moderator intervention
  • v0.3.7 - PR #301439 (opened Oct 6) - Eventually merged after moderator intervention
  • v0.3.9 - PR #304230 (opened Oct 16) - Currently blocked
  • v0.3.11 - PR #304550 (opened Oct 17) - Currently blocked with Internal-Error-Dynamic-Scan, Needs-Attention labels

Pipeline Issues

  • PRs are showing Internal-Error-Dynamic-Scan and Retry-1 labels
  • Pipeline automation keeps removing and re-adding assignees
  • No helpful error messages in pipeline logs
  • Same reviewer assigned across multiple stuck PRs

Root Cause Analysis

The Windows CLI executable (thv.exe) is not Authenticode signed, unlike the Windows UI installer which does have proper code signing. Unsigned executables are more likely to trigger Windows Defender false positives during WinGet's automated security scanning.

Proposed Solution

Add Authenticode (DigiCert) signing to the CLI Windows executable in the release build pipeline. Similar to how we do it with the ToolHive Studio

Current State

  • Windows UI installer: ✅ Signed with DigiCert
  • Windows CLI executable: ❌ Not signed (may have Sigstore signing, but that doesn't help with Windows Defender)
  • The signing was likely added after the initial GoReleaser setup, so it was missed for the CLI

Impact

  • Windows users cannot install the latest CLI versions via WinGet
  • Latest available version stuck at v0.3.8 (3 releases behind)
  • Growing backlog of open PRs in microsoft/winget-pkgs
  • Users must use alternative installation methods (direct download, Homebrew on WSL)

References

  • Internal issue: stacklok/toolhive#2100
  • WinGet moderator comment: https://github.com/stacklok/toolhive/issues/2100#issuecomment-...
  • WinGet PR queue showing pattern of delays

ChrisJBurns avatar Oct 21 '25 14:10 ChrisJBurns

Solution: Cross-Platform Windows Code Signing with jsign + DigiCert KeyLocker

Good news - we can sign Windows executables in the existing GoReleaser workflow without breaking up the build process or switching to Windows runners!

How It Works

jsign is a cross-platform (Java-based) code signing tool that:

  • ✅ Runs on Linux runners (no need to split the build matrix)
  • ✅ Natively supports DigiCert KeyLocker/ONE (our existing cert provider)
  • ✅ Integrates directly into GoReleaser via archive hooks
  • ✅ Signs binaries before they're zipped for WinGet
  • ✅ Uses the same DigiCert credentials we already have for toolhive-studio

Required Changes

1. Update .goreleaser.yaml

Add a hooks.before section to the archives config to sign Windows binaries after build but before archiving:

# This section defines the release format.
archives:
  - formats: [ 'tar.gz' ]
    name_template: "{{ .ProjectName }}_{{ .Version }}_{{ .Os }}_{{ .Arch }}"
    format_overrides:
      - goos: windows
        formats: [ 'zip' ]
    hooks:
      before:
        # Sign Windows executables with DigiCert KeyLocker before archiving
        - sh -c 'if [ "{{ .Os }}" = "windows" ]; then jsign --storetype DIGICERTONE --storepass "${SM_API_KEY}|${SM_CLIENT_CERT_FILE}|${SM_CLIENT_CERT_PASSWORD}" --alias "${SM_CODE_SIGNING_CERT_SHA1_HASH}" --tsaurl http://timestamp.digicert.com --tsmode RFC3161 --name "ToolHive" --url "https://stacklok.com" "{{ .Path }}"; fi'

2. Update .github/workflows/releaser.yml

Add these steps before the "Run GoReleaser" step:

      - name: Setup Java
        uses: actions/setup-java@v4
        with:
          distribution: 'temurin'
          java-version: '17'

      - name: Install jsign
        run: |
          wget -q https://github.com/ebourg/jsign/releases/download/7.0/jsign-7.0.jar
          sudo mkdir -p /opt/jsign
          sudo mv jsign-7.0.jar /opt/jsign/
          echo '#!/bin/sh' | sudo tee /usr/local/bin/jsign
          echo "exec java -jar /opt/jsign/jsign-7.0.jar \"\$@\"" | sudo tee -a /usr/local/bin/jsign
          sudo chmod +x /usr/local/bin/jsign
          jsign --version

      - name: Setup DigiCert Code Signing
        env:
          SM_CLIENT_CERT_FILE_B64: ${{ secrets.SM_CLIENT_CERT_FILE_B64 }}
        run: |
          # Decode client certificate from base64
          echo "$SM_CLIENT_CERT_FILE_B64" | base64 -d > /tmp/digicert_client.p12
          echo "SM_CLIENT_CERT_FILE=/tmp/digicert_client.p12" >> $GITHUB_ENV

      - name: Run GoReleaser
        id: run-goreleaser
        uses: goreleaser/goreleaser-action@e435ccd777264be153ace6237001ef4d979d3a7a # v6
        with:
          distribution: goreleaser
          version: "~> v2"
          args: release --clean
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          WINGET_GITHUB_TOKEN: ${{ secrets.WINGET_GITHUB_TOKEN }}
          HOMEBREW_TAP_GITHUB_TOKEN: ${{ secrets.HOMEBREW_TAP_GITHUB_TOKEN }}
          VERSION: ${{ needs.compute-build-flags.outputs.version }}
          COMMIT: ${{ needs.compute-build-flags.outputs.commit }}
          COMMIT_DATE: ${{ needs.compute-build-flags.outputs.commit-date }}
          TREE_STATE: ${{ needs.compute-build-flags.outputs.tree-state }}
          # DigiCert KeyLocker credentials (same as toolhive-studio)
          SM_API_KEY: ${{ secrets.SM_API_KEY }}
          SM_CLIENT_CERT_FILE: /tmp/digicert_client.p12
          SM_CLIENT_CERT_PASSWORD: ${{ secrets.SM_CLIENT_CERT_PASSWORD }}
          SM_CODE_SIGNING_CERT_SHA1_HASH: ${{ secrets.SM_CODE_SIGNING_CERT_SHA1_HASH }}

Required GitHub Secrets

These secrets should already exist in the repository (currently used by toolhive-studio):

  • SM_API_KEY - DigiCert API key
  • SM_CLIENT_CERT_FILE_B64 - Base64-encoded PKCS#12 client certificate
  • SM_CLIENT_CERT_PASSWORD - Client certificate password
  • SM_CODE_SIGNING_CERT_SHA1_HASH - Certificate thumbprint/alias

How This Solves The Issue

  1. Windows Defender Detection: Authenticode-signed executables are trusted by Windows Defender and WinGet's automated security scanning
  2. No Build Split: Everything runs on Linux runners in a single unified build process
  3. Same Certificate: Uses the identical DigiCert KeyLocker setup as toolhive-studio
  4. Timestamping: Includes RFC3161 timestamping so signatures remain valid after the certificate expires
  5. WinGet Compatible: Binaries are signed before being zipped, so WinGet receives signed executables

Testing the Solution

After implementing, verify the signature with:

# On Windows
signtool verify /pa /v thv.exe

# Should show:
# - "Successfully verified: thv.exe"
# - Certificate issued by DigiCert
# - Timestamping information

References

  • jsign DigiCert ONE docs: https://ebourg.github.io/jsign/#digicert-one
  • toolhive-studio signing implementation: https://github.com/stacklok/toolhive-studio/blob/main/.github/actions/setup-windows-codesign/action.yml
  • GoReleaser hooks documentation: https://goreleaser.com/customization/archive/#packaging-only-the-binaries

danbarr avatar Nov 26 '25 04:11 danbarr