`cardano-ledger-shelley-ma-test` failures in nightly CI

[neilmayhew]

Reproducible with:

NIGHTLY=true cabal test cardano-ledger-shelley-ma-test \
  --test-options='-p "/Only valid CHAIN STS signals are generated/"' \
  --test-options='--quickcheck-replay="(SMGen 5277290757380433267 7778229362812927015,18)"'

Output:

ShelleyMA Ledger - nightly
  Allegra Ledger - nightly
    Trace generators properties
      Only valid CHAIN STS signals are generated: FAIL (28.21s)
        *** Failed! (after 1 test and 1 shrink):
        Exception while generating shrink-list:
          
          
          AssertionViolation (ShelleyLEDGER (AllegraEra StandardCrypto))
          
            Deposit pot must equal obligation (LEDGER)
          
          CERTS
          
          (slot,keyDeposit,poolDeposit) (SlotNo 685,Coin 16,Coin 23)
          The Pot (utxosDeposited) = Coin 1660
          Total Obligations = Coin 1644
             Stake deposits = Coin 816
             Pool deposits = Coin 828
             DRep deposits = Coin 0
             Proposal deposits = Coin 0
          
          Consumed = Consumed(Inputs 0, Refunds 0, Withdrawals 0) = 0
          Produced = Produced(Outputs 2576711589, Fees 969865, Deposits 0) = 2577681454
        Exception thrown while showing test case:
          
          
          AssertionViolation (ShelleyLEDGER (AllegraEra StandardCrypto))
          
            Deposit pot must equal obligation (LEDGER)
          
          CERTS
          
          (slot,keyDeposit,poolDeposit) (SlotNo 685,Coin 16,Coin 23)
          The Pot (utxosDeposited) = Coin 1660
          Total Obligations = Coin 1644
             Stake deposits = Coin 816
             Pool deposits = Coin 828
             DRep deposits = Coin 0
             Proposal deposits = Coin 0
          
          Consumed = Consumed(Inputs 0, Refunds 0, Withdrawals 0) = 0
          Produced = Produced(Outputs 2576711589, Fees 969865, Deposits 0) = 2577681454

[neilmayhew]

A git bisect identified cf0faa89 as the origin of this problem. This changed the bounds on QuickCheck from ^>=2.14 to >=2.14.

[neilmayhew]

The test failure can be exposed by disabling shrinking in forAllTraceFromInitState:

--- a/libs/small-steps/testlib/Test/Control/State/Transition/Trace/Generator/QuickCheck.hs
+++ b/libs/small-steps/testlib/Test/Control/State/Transition/Trace/Generator/QuickCheck.hs
@@ -178,7 +178,7 @@ forAllTraceFromInitState ::
 forAllTraceFromInitState baseEnv maxTraceLength traceGenEnv genSt0 prop =
   QuickCheck.forAllShrinkBlind
     (traceFromInitState @sts @traceGenEnv baseEnv maxTraceLength traceGenEnv genSt0)
-    (shrinkTrace @sts @traceGenEnv baseEnv)
+    (const [])
     prop
 
 -- | Check a property on the 'sts' traces.

This shows the failure to be:

RealChainPredicateFailure (BlockSizeTooLargeCHAIN 61252 54298) :| []

It's not clear to me why the change in QuickCheck version would increase block sizes. It could be that it's just not reproducible with this seed using prior versions, but we weren't getting this specific nightly test failure until relatively recently, so perhaps there's some additional cause.

[neilmayhew]

This test failure seems to be an outlier. The instance in the nightly run from 2024-10-24 is the only occurrence since the earliest date we have logs for (2024-08-01).

I believe the bisect identified the QuickCheck version change simply because that changed the seed needed to produce a given set of test inputs, and the seed we have was taken after the version change.

I think we should ignore this for now and come back to it if we see this failure again.

[neilmayhew]

Reopening to preempt this happening again by adding a discard

[lehins]

Another reproducer from CI:

      Use --quickcheck-replay="(SMGen 9691748105170960432 18368457056822691145,71)" to reproduce.
      Use -p '/Only valid CHAIN STS signals are generated/' to rerun this test only.

[lehins]

One more reproducer in cardano-ledger-shelley:

Only valid CHAIN STS signals are generated:                                FAIL (114.84s)
    *** Failed! (after 74 tests and 1 shrink):
    Exception while generating shrink-list:
      
      
      AssertionViolation (ShelleyLEDGER ShelleyEra)
      
        Deposit pot must equal obligation (LEDGER)
      
      CERTS
        ShelleyTxCertDelegCert (ShelleyDelegCert (ScriptHashObj (ScriptHash "75f59830fcbc18aa798dfb33170e[56](https://github.com/IntersectMBO/cardano-ledger/actions/runs/13683219837/job/38263547760?pr=4906#step:14:57)6a008b0e75c199b806c8afb2f4")) (KeyHash {unKeyHash = "c3547689a6c96f976a92d1b6388316e65b2ec897de1be4fd80672565"}))
      
      (slot,keyDeposit,poolDeposit) (SlotNo 30,Coin 6,Coin 98)
      The Pot (utxosDeposited) = Coin 526
      Total Obligations = Coin 520
         Stake deposits = Coin 30
         Pool deposits = Coin 490
         DRep deposits = Coin 0
         Proposal deposits = Coin 0
      
      Consumed = Consumed(Inputs 0, Refunds 0, Withdrawals 0) = 0
      Produced = Produced(Outputs 2[59](https://github.com/IntersectMBO/cardano-ledger/actions/runs/13683219837/job/38263547760?pr=4906#step:14:60)417252, Fees 722448, Deposits 0) = 2[60](https://github.com/IntersectMBO/cardano-ledger/actions/runs/13683219837/job/38263547760?pr=4906#step:14:61)139700
    Exception thrown while showing test case:
      
      
      AssertionViolation (ShelleyLEDGER ShelleyEra)
      
        Deposit pot must equal obligation (LEDGER)
      
      CERTS
        ShelleyTxCertDelegCert (ShelleyDelegCert (ScriptHashObj (ScriptHash "75f59830fcbc18aa798dfb33170e566a008b0e75c199b806c8afb2f4")) (KeyHash {unKeyHash = "c3547689a6c96f976a92d1b6388316e65b2ec897de1be4fd80672565"}))
      
      (slot,keyDeposit,poolDeposit) (SlotNo 30,Coin 6,Coin 98)
      The Pot (utxosDeposited) = Coin 526
      Total Obligations = Coin 520
         Stake deposits = Coin 30
         Pool deposits = Coin 490
         DRep deposits = Coin 0
         Proposal deposits = Coin 0
      
      Consumed = Consumed(Inputs 0, Refunds 0, Withdrawals 0) = 0
      Produced = Produced(Outputs 259417252, Fees 722448, Deposits 0) = 260139700
    
    Use --quickcheck-replay="(SMGen 3814140053751207545 15493553050[61](https://github.com/IntersectMBO/cardano-ledger/actions/runs/13683219837/job/38263547760?pr=4906#step:14:62)9018659,73)" to reproduce.
    Use -p '/Only valid CHAIN STS signals are generated/' to rerun this test only.