Write a CI script for checking that the project can be built with all compiler versions

[PaulRBerg]

This would help prevent issues like https://github.com/paulrberg/prb-test/issues/5 from happening again.

[sambacha]

question: what about doing an ephemeral pragma version and using git tag's to reference the compiler version?

i.e.

pragma solidity *;

git ls-remote -t --refs https://github.com/paulrberg/prb-test | sed -E 's/^[[:xdigit:]]+[[:space:]]+refs\/tags\/(.+)/\1/g'

gives:

v0.1.0
v0.1.1
v0.1.2

so you would. have to tag the compiler tests in a different way. If you don't want to do tags, you could do something like this below, but will still need ephemeral solidity versioning to make it easy:

Here is a shell script and an optional artifactor to create test results potentially about the run itself (this is more relevant for ensuring the CI behavior etc than testing the contracts themselves)

Shell script

GitHub Actions workflow script

#!/usr/bin/env zsh
# NOTE: using zsh because easier to do this in than bash
# ensure env
export LANG='en_US.UTF-8'
export LANGUAGE='en_US:en'
export LC_ALL='en_US.UTF-8'

# optional env configuration;
export DAPP_TEST_FUZZ_RUNS=10000
export DAPP_LINK_TEST_LIBRARIES=0
export DAPP_TEST_VERBOSITY=1
export DAPP_TEST_SMTTIMEOUT=500000

# solidity versions to test
# see https://github.com/sambacha/dappspec/blob/master/abi/src/Accretive_Versioning.md#solidity-versions-and-feature-sets
solcVersions=(
  '0.7.0'
  '0.6.12'
  '0.6.3'
  '0.5.17'
  '0.8.3'
  '0.8.7'
  '0.8.15'
)

# $1 : version
# shellcheck disable=SC2016
function writeFoundryConfig() {
  cat << EOF > solc-test.sh
export DAPP_SOLC_VERSION=: "$1"
EOF
}

# Example
# create output file artifact
csvOutput="SolidityVersionAnalysis.csv"
echo "Solidity Version", "name", "constant", "payable", "stateMutability", "type" > $csvOutput
echo "" >> $csvOutput
sleep 1

# clear output of color encoding;
# forge $cmd | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" | tee $outputFile
for version in $solcVersions; do
  writeFoundryConfig $version
  echo $(forge build --force; forge test -vvv; forge snapshot --allow-failures >02_gas_report)
  # parseArtifact is optional step, just an example
  node parseArtifact $version >> $csvOutput
done

#  parseArtifact 
# sanity test; optional
# ETHERS_SOLC_LOG=in=in.json,out=out.json;  forge build --force

parseArtifact.js

// parseArtifact.js
const fs = require('fs')

const data = fs.readFileSync("./out${CONTRACT}/${CONTRACT}.json")
const { abi } = JSON.parse (data.toString())

const solidityVersion = process.argv[2]
const keys = ["name", "constant", "payable", "stateMutability", "type"]

for (let entry of abi) {
  if (entry.type !== 'function') continue
  const row = [solidityVersion]
    .concat(keys.map(k => entry[k] || 'undefined'))
  console.log(row.join(', '))
};

[sambacha]

posting this I realize you can just use the $1 : version and pass that to forge as long as you had in the foundry.toml file disabling auto detect, avoiding the writeFoundryConfig function entirely

[PaulRBerg]

Interesting idea @sambacha, but I think it would be easier to just run foundry build iteratively between a min and a max version. Here's a dummy GitHub Action shell script that I wrote for this:

#!/usr/bin/env bash

set -euo pipefail

# check that the minimum Solidity version has been supplied
if [ -z "${1-}" ]; then
  echo "No minimum Solidity version supplied"
  exit 1
fi

# check that the maximum Solidity version has been supplied
if [ -z "${2-}" ]; then
  echo "No maximum Solidity version supplied"
  exit 1
fi

# name the input arguments
min=$1
max=$2

# the Solidity versions supported by Forge
declare -a supported_versions=(
  "0.5.0" "0.5.1" "0.5.2" "0.5.3" "0.5.4" "0.5.5" "0.5.6" "0.5.7" "0.5.8" "0.5.9" "0.5.10" "0.5.11" "0.5.12" "0.5.13" "0.5.14" "0.5.15" "0.5.16" "0.5.17"
  "0.6.0" "0.6.1" "0.6.2" "0.6.3" "0.6.4" "0.6.5" "0.6.6" "0.6.7" "0.6.8" "0.6.9" "0.6.10" "0.6.11" "0.6.12"
  "0.7.0" "0.7.1" "0.7.2" "0.7.3" "0.7.4" "0.7.5" "0.7.6"
  "0.8.0" "0.8.1" "0.8.2" "0.8.3" "0.8.4" "0.8.5" "0.8.6" "0.8.7" "0.8.8" "0.8.9" "0.8.10" "0.8.11" "0.8.12" "0.8.13" "0.8.14" "0.8.15"
)

# check if the minimum Solidity version supplied by user is among the supported versions
if [[ ! " ${supported_versions[*]} " =~ " $min " ]]; then
    echo "Minimum Solidity version not supported"
    exit 1
fi

# check if the maximum Solidity version supplied by user is among the supported versions
if [[ ! " ${supported_versions[*]} " =~ " $max " ]]; then
    echo "Minimum Solidity version not supported"
    exit 1
fi

# get the index of the minimum version
for mini in "${!supported_versions[@]}"; do
   [[ "${supported_versions[$mini]}" = "$min" ]] && break
done

# get the index of the maximum version
for maxi in "${!supported_versions[@]}"; do
   [[ "${supported_versions[$maxi]}" = "$max" ]] && break
done

# run "forge build" over
for i in $(seq $mini $maxi); do
  forge build --use ${supported_versions[$i]}
done

[sambacha]

Interesting idea @sambacha, but I think it would be easier to just run foundry build iteratively between a min and a max version. Here's a dummy GitHub Action shell script that I wrote for this:

#!/usr/bin/env bash

set -euo pipefail

# check that the minimum Solidity version has been supplied
if [ -z "${1-}" ]; then
  echo "No minimum Solidity version supplied"
  exit 1
fi

# check that the maximum Solidity version has been supplied
if [ -z "${2-}" ]; then
  echo "No maximum Solidity version supplied"
  exit 1
fi

# name the input arguments
min=$1
max=$2

# the Solidity versions supported by Forge
declare -a supported_versions=(
  "0.5.0" "0.5.1" "0.5.2" "0.5.3" "0.5.4" "0.5.5" "0.5.6" "0.5.7" "0.5.8" "0.5.9" "0.5.10" "0.5.11" "0.5.12" "0.5.13" "0.5.14" "0.5.15" "0.5.16" "0.5.17"
  "0.6.0" "0.6.1" "0.6.2" "0.6.3" "0.6.4" "0.6.5" "0.6.6" "0.6.7" "0.6.8" "0.6.9" "0.6.10" "0.6.11" "0.6.12"
  "0.7.0" "0.7.1" "0.7.2" "0.7.3" "0.7.4" "0.7.5" "0.7.6"
  "0.8.0" "0.8.1" "0.8.2" "0.8.3" "0.8.4" "0.8.5" "0.8.6" "0.8.7" "0.8.8" "0.8.9" "0.8.10" "0.8.11" "0.8.12" "0.8.13" "0.8.14" "0.8.15"
)

# check if the minimum Solidity version supplied by user is among the supported versions
if [[ ! " ${supported_versions[*]} " =~ " $min " ]]; then
    echo "Minimum Solidity version not supported"
    exit 1
fi

# check if the maximum Solidity version supplied by user is among the supported versions
if [[ ! " ${supported_versions[*]} " =~ " $max " ]]; then
    echo "Minimum Solidity version not supported"
    exit 1
fi

# get the index of the minimum version
for mini in "${!supported_versions[@]}"; do
   [[ "${supported_versions[$mini]}" = "$min" ]] && break
done

# get the index of the maximum version
for maxi in "${!supported_versions[@]}"; do
   [[ "${supported_versions[$maxi]}" = "$max" ]] && break
done

# run "forge build" over
for i in $(seq $mini $maxi); do
  forge build --use ${supported_versions[$i]}
done

Additionally, we would want to ensure that the invoked commands properly exit, and run in parallel

# Invoke the provided commands in parallel and collect their exit codes.
pids=()
for userCommand in "${userCommands[@]}"; do
  eval "$userCommand" & pids+=($!)
done

# If any one of the invoked commands exited with a non-zero exit code, exit the whole thing with code 1.
for pid in "${pids[@]}"; do
  if ! wait "$pid"; then
    exit 1
  fi
done

# All the invoked commands must have exited with code zero.
exit 0

Yea I like what you have much more as you can put in the actions workflow file a workflow dispatch where you can enter the versions you want tested as opposed to running against all. very nice.

https://github.com/sambacha/bash-action