cardano-ledger-shelley-ma-test failures in nightly CI

[neilmayhew]

Reproducible with:

NIGHTLY=true cabal test cardano-ledger-shelley-ma-test \
  --test-options='-p "/Only valid CHAIN STS signals are generated/"'
  --test-options='--quickcheck-replay="(SMGen 5277290757380433267 7778229362812927015,18)"'

Output:

ShelleyMA Ledger - nightly
  Allegra Ledger - nightly
    Trace generators properties
      Only valid CHAIN STS signals are generated: FAIL (28.21s)
        *** Failed! (after 1 test and 1 shrink):
        Exception while generating shrink-list:
          
          
          AssertionViolation (ShelleyLEDGER (AllegraEra StandardCrypto))
          
            Deposit pot must equal obligation (LEDGER)
          
          CERTS
          
          (slot,keyDeposit,poolDeposit) (SlotNo 685,Coin 16,Coin 23)
          The Pot (utxosDeposited) = Coin 1660
          Total Obligations = Coin 1644
             Stake deposits = Coin 816
             Pool deposits = Coin 828
             DRep deposits = Coin 0
             Proposal deposits = Coin 0
          
          Consumed = Consumed(Inputs 0, Refunds 0, Withdrawals 0) = 0
          Produced = Produced(Outputs 2576711589, Fees 969865, Deposits 0) = 2577681454
        Exception thrown while showing test case:
          
          
          AssertionViolation (ShelleyLEDGER (AllegraEra StandardCrypto))
          
            Deposit pot must equal obligation (LEDGER)
          
          CERTS
          
          (slot,keyDeposit,poolDeposit) (SlotNo 685,Coin 16,Coin 23)
          The Pot (utxosDeposited) = Coin 1660
          Total Obligations = Coin 1644
             Stake deposits = Coin 816
             Pool deposits = Coin 828
             DRep deposits = Coin 0
             Proposal deposits = Coin 0
          
          Consumed = Consumed(Inputs 0, Refunds 0, Withdrawals 0) = 0
          Produced = Produced(Outputs 2576711589, Fees 969865, Deposits 0) = 2577681454

[neilmayhew]

A git bisect identified cf0faa8 as the origin of this problem. This changed the bounds on QuickCheck from ^>=2.14 to >=2.14.

[neilmayhew]

The test failure can be exposed by disabling shrinking in forAllTraceFromInitState:

--- a/libs/small-steps/testlib/Test/Control/State/Transition/Trace/Generator/QuickCheck.hs
+++ b/libs/small-steps/testlib/Test/Control/State/Transition/Trace/Generator/QuickCheck.hs
@@ -178,7 +178,7 @@ forAllTraceFromInitState ::
 forAllTraceFromInitState baseEnv maxTraceLength traceGenEnv genSt0 prop =
   QuickCheck.forAllShrinkBlind
     (traceFromInitState @sts @traceGenEnv baseEnv maxTraceLength traceGenEnv genSt0)
-    (shrinkTrace @sts @traceGenEnv baseEnv)
+    (const [])
     prop
 
 -- | Check a property on the 'sts' traces.

This shows the failure to be:

RealChainPredicateFailure (BlockSizeTooLargeCHAIN 61252 54298) :| []

It's not clear to me why the change in QuickCheck version would increase block sizes. It could be that it's just not reproducible with this seed using prior versions, but we weren't getting this specific nightly test failure until relatively recently, so perhaps there's some additional cause.

[neilmayhew]

This test failure seems to be an outlier. The instance in the nightly run from 2024-10-24 is the only occurrence since the earliest date we have logs for (2024-08-01).

I believe the bisect identified the QuickCheck version change simply because that changed the seed needed to produce a given set of test inputs, and the seed we have was taken after the version change.

I think we should ignore this for now and come back to it if we see this failure again.

[neilmayhew]

Reopening to preempt this happening again by adding a discard