ouroboros-network icon indicating copy to clipboard operation
ouroboros-network copied to clipboard

ChainDB q-s-m failure with `GetIsValid`

Open amesgen opened this issue 2 years ago • 2 comments

To reproduce

Run

cabal run test-storage -- -p 'ChainDB q-s-m' --quickcheck-replay=455411

on fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa (master at time of writing). This yields

(PredicateC (Resp {getResp = Right (IsValid (IsValidResult {real = True, isValid = Just True}))}

:/=          Resp {getResp = Right (IsValid (IsValidResult {real = False, isValid = Nothing}))}))

This is exactly what #3651 ought to have fixed.

Cursory glance

The QSM commands look like this:

 - Add valid   block Genesis >: A with hash 2173152789646624309
 - Add valid   block A       >: B with hash 9122512639099877219
 - Add valid   block B       >: C with hash -7915818155603348863
 - Add invalid block C       >: D with hash -1393368485677491682
 - Add valid   block A       >: E with hash -2651776708921877814
 - Add invalid block E       >: F with hash 128720366063791916
 - Check if E is valid
    - ChainSel says: yes, valid
    - Model says:    unknown

Chain tree:
  Genesis >: A >: B >: C  :> D*
                  ∧    ∧
               >: E >: F*
* invalid

My interpretation: The model is wrong here, we have to validate E as the chain diff with rollback 2 and fragment E >: F is an improvement over our currently selected chain A >: B >: C as C < F. So even though we end up not selecting it as F is invalid, ChainSel still learns that E is valid.

We already have some asymmetry when comparing these validity results:

https://github.com/input-output-hk/ouroboros-network/blob/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa/ouroboros-consensus-test/test-storage/Test/Ouroboros/Storage/ChainDB/StateMachine.hs#L501-L504

Concretely, it is for the "other way around": when ChainSel does not know whether a block is valid, we intentionally don't check whether the model agrees.

Full log

Click to expand
      ChainDB q-s-m
        sequential:
Model {
  dbModel = Model {
    volatileDbBlocks = Map.fromList
      [],
    immutableDbChain = Genesis,
    cps = ChainProducerState {
      chainState = Genesis,
      chainFollowers = Map.fromList
        [],
      nextFollowerId = 0},
    currentLedger = ExtLedgerState {
      ledgerState = TestLedger {
        lastAppliedPoint = Point Origin,
        lastAppliedHash = GenesisHash},
      headerState = HeaderState {
        headerStateTip = Origin,
        headerStateChainDep = `()`}},
    initLedger = ExtLedgerState {
      ledgerState = TestLedger {
        lastAppliedPoint = Point Origin,
        lastAppliedHash = GenesisHash},
      headerState = HeaderState {
        headerStateTip = Origin,
        headerStateChainDep = `()`}},
    iterators = Map.fromList [],
    valid = Set.fromList [],
    invalid = Map.fromList [],
    currentSlot = SlotNo 0,
    maxClockSkew = 100000,
    isOpen = True},
  knownIters = RefEnv [],
  knownFollowers = RefEnv [],
  modelConfig = Opaque}

   == At
  { unAt =
      AddBlock
        TestBlock
          { testHeader =
              TestHeader
                { thHash = TestHeaderHash 2173152789646624309
                , thPrevHash = GenesisHash
                , thBodyHash = TestBodyHash 590680769285548757
                , thSlotNo = SlotNo 0
                , thBlockNo = BlockNo 0
                , thChainLength = ChainLength 1
                , thIsEBB = EBB (EpochNo 0)
                }
          , testBody = TestBody { tbForkNo = 3 , tbIsValid = True }
          }
  } ==> At
  { unAt =
      Resp
        { getResp =
            Right
              (Point
                 (At
                    Block
                      { blockPointSlot = SlotNo 0
                      , blockPointHash = TestHeaderHash 2173152789646624309
                      }))
        }
  } [ 0 ]

Model {
  dbModel = Model {
    volatileDbBlocks = Map.fromList
      [
        +_×_
          (TestHeaderHash
            2173152789646624309)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                2173152789646624309,
              thPrevHash = GenesisHash,
              thBodyHash = TestBodyHash
                590680769285548757,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 0,
              thChainLength = ChainLength 1,
              thIsEBB = EBB (EpochNo 0)},
            testBody = TestBody {
              tbForkNo = 3,
              tbIsValid = True}}],
    immutableDbChain = Genesis,
    cps = ChainProducerState {
      chainState = -Genesis
      +:>
        Genesis
        TestBlock {
          testHeader = TestHeader {
            thHash = TestHeaderHash
              2173152789646624309,
            thPrevHash = GenesisHash,
            thBodyHash = TestBodyHash
              590680769285548757,
            thSlotNo = SlotNo 0,
            thBlockNo = BlockNo 0,
            thChainLength = ChainLength 1,
            thIsEBB = EBB (EpochNo 0)},
          testBody = TestBody {
            tbForkNo = 3,
            tbIsValid = True}},
      nextFollowerId = 0,
      chainFollowers = ...},
    currentLedger = ExtLedgerState {
      ledgerState = TestLedger {
        lastAppliedPoint = Point
          -Origin
          +(At
            Block {
              blockPointSlot = SlotNo 0,
              blockPointHash = TestHeaderHash
                2173152789646624309}),
        lastAppliedHash = -GenesisHash
        +BlockHash
          (TestHeaderHash
            2173152789646624309)},
      headerState = HeaderState {
        headerStateTip = -Origin
        +At
          AnnTip {
            annTipSlotNo = SlotNo 0,
            annTipBlockNo = BlockNo 0,
            annTipInfo = TipInfoIsEBB
              (TestHeaderHash
                2173152789646624309)
              IsEBB},
        headerStateChainDep = `()`}},
    valid = Set.fromList
      [
        +TestHeaderHash
          2173152789646624309],
    maxClockSkew = 100000,
    isOpen = True,
    initLedger = ...},
  modelConfig = Opaque,
  knownIters = ...}

   == At
  { unAt =
      AddBlock
        TestBlock
          { testHeader =
              TestHeader
                { thHash = TestHeaderHash 9122512639099877219
                , thPrevHash = BlockHash (TestHeaderHash 2173152789646624309)
                , thBodyHash = TestBodyHash 590681868797176966
                , thSlotNo = SlotNo 0
                , thBlockNo = BlockNo 1
                , thChainLength = ChainLength 2
                , thIsEBB = RegularBlock
                }
          , testBody = TestBody { tbForkNo = 2 , tbIsValid = True }
          }
  } ==> At
  { unAt =
      Resp
        { getResp =
            Right
              (Point
                 (At
                    Block
                      { blockPointSlot = SlotNo 0
                      , blockPointHash = TestHeaderHash 9122512639099877219
                      }))
        }
  } [ 0 ]

Model {
  dbModel = Model {
    volatileDbBlocks = Map.fromList
      [
        _×_
          (TestHeaderHash
            2173152789646624309)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                2173152789646624309,
              thPrevHash = GenesisHash,
              thBodyHash = TestBodyHash
                590680769285548757,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 0,
              thChainLength = ChainLength 1,
              thIsEBB = EBB (EpochNo 0)},
            testBody = TestBody {
              tbForkNo = 3,
              tbIsValid = True}},
        +_×_
          (TestHeaderHash
            9122512639099877219)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                9122512639099877219,
              thPrevHash = BlockHash
                (TestHeaderHash
                  2173152789646624309),
              thBodyHash = TestBodyHash
                590681868797176966,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 1,
              thChainLength = ChainLength 2,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 2,
              tbIsValid = True}}],
    immutableDbChain = Genesis,
    cps = ChainProducerState {
      chainState = :>
        -Genesis
        +(:>
          Genesis
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                2173152789646624309,
              thPrevHash = GenesisHash,
              thBodyHash = TestBodyHash
                590680769285548757,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 0,
              thChainLength = ChainLength 1,
              thIsEBB = EBB (EpochNo 0)},
            testBody = TestBody {
              tbForkNo = 3,
              tbIsValid = True}})
        TestBlock {
          testHeader = TestHeader {
            thHash = TestHeaderHash
              -2173152789646624309
              +9122512639099877219,
            thPrevHash = -GenesisHash
            +BlockHash
              (TestHeaderHash
                2173152789646624309),
            thBodyHash = TestBodyHash
              -590680769285548757
              +590681868797176966,
            thBlockNo = BlockNo -0 +1,
            thChainLength = ChainLength
              -1
              +2,
            thIsEBB = -EBB (EpochNo 0)
            +RegularBlock,
            thSlotNo = ...},
          testBody = TestBody {
            tbForkNo = -3 +2,
            tbIsValid = True}},
      nextFollowerId = 0,
      chainFollowers = ...},
    currentLedger = ExtLedgerState {
      ledgerState = TestLedger {
        lastAppliedPoint = Point
          (At
            Block {
              blockPointHash = TestHeaderHash
                -2173152789646624309
                +9122512639099877219,
              blockPointSlot = ...}),
        lastAppliedHash = BlockHash
          (TestHeaderHash
            -2173152789646624309
            +9122512639099877219)},
      headerState = HeaderState {
        headerStateTip = At
          AnnTip {
            annTipBlockNo = BlockNo -0 +1,
            annTipInfo = TipInfoIsEBB
              (TestHeaderHash
                -2173152789646624309
                +9122512639099877219)
              -IsEBB
              +IsNotEBB,
            annTipSlotNo = ...},
        headerStateChainDep = `()`}},
    valid = Set.fromList
      [
        TestHeaderHash
          2173152789646624309,
        +TestHeaderHash
          9122512639099877219],
    maxClockSkew = 100000,
    isOpen = True,
    initLedger = ...},
  modelConfig = Opaque,
  knownIters = ...}

   == At
  { unAt =
      AddBlock
        TestBlock
          { testHeader =
              TestHeader
                { thHash = TestHeaderHash (-7915818155603348863)
                , thPrevHash = BlockHash (TestHeaderHash 9122512639099877219)
                , thBodyHash = TestBodyHash 590680769285548757
                , thSlotNo = SlotNo 1
                , thBlockNo = BlockNo 2
                , thChainLength = ChainLength 3
                , thIsEBB = RegularBlock
                }
          , testBody = TestBody { tbForkNo = 3 , tbIsValid = True }
          }
  } ==> At
  { unAt =
      Resp
        { getResp =
            Right
              (Point
                 (At
                    Block
                      { blockPointSlot = SlotNo 1
                      , blockPointHash = TestHeaderHash (-7915818155603348863)
                      }))
        }
  } [ 0 ]

Model {
  dbModel = Model {
    volatileDbBlocks = Map.fromList
      [
        +_×_
          (TestHeaderHash
            `-7915818155603348863`)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                `-7915818155603348863`,
              thPrevHash = BlockHash
                (TestHeaderHash
                  9122512639099877219),
              thBodyHash = TestBodyHash
                590680769285548757,
              thSlotNo = SlotNo 1,
              thBlockNo = BlockNo 2,
              thChainLength = ChainLength 3,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 3,
              tbIsValid = True}},
        _×_
          (TestHeaderHash
            2173152789646624309)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                2173152789646624309,
              thPrevHash = GenesisHash,
              thBodyHash = TestBodyHash
                590680769285548757,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 0,
              thChainLength = ChainLength 1,
              thIsEBB = EBB (EpochNo 0)},
            testBody = TestBody {
              tbForkNo = 3,
              tbIsValid = True}},
        _×_
          (TestHeaderHash
            9122512639099877219)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                9122512639099877219,
              thPrevHash = BlockHash
                (TestHeaderHash
                  2173152789646624309),
              thBodyHash = TestBodyHash
                590681868797176966,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 1,
              thChainLength = ChainLength 2,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 2,
              tbIsValid = True}}],
    immutableDbChain = Genesis,
    cps = ChainProducerState {
      chainState = :>
        (:>
          -Genesis
          +(:>
            Genesis
            TestBlock {
              testHeader = TestHeader {
                thHash = TestHeaderHash
                  2173152789646624309,
                thPrevHash = GenesisHash,
                thBodyHash = TestBodyHash
                  590680769285548757,
                thSlotNo = SlotNo 0,
                thBlockNo = BlockNo 0,
                thChainLength = ChainLength 1,
                thIsEBB = EBB (EpochNo 0)},
              testBody = TestBody {
                tbForkNo = 3,
                tbIsValid = True}})
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                -2173152789646624309
                +9122512639099877219,
              thPrevHash = -GenesisHash
              +BlockHash
                (TestHeaderHash
                  2173152789646624309),
              thBodyHash = TestBodyHash
                -590680769285548757
                +590681868797176966,
              thBlockNo = BlockNo -0 +1,
              thChainLength = ChainLength
                -1
                +2,
              thIsEBB = -EBB (EpochNo 0)
              +RegularBlock,
              thSlotNo = ...},
            testBody = TestBody {
              tbForkNo = -3 +2,
              tbIsValid = True}})
        TestBlock {
          testHeader = TestHeader {
            thHash = TestHeaderHash
              -9122512639099877219
              +`-7915818155603348863`,
            thPrevHash = BlockHash
              (TestHeaderHash
                -2173152789646624309
                +9122512639099877219),
            thBodyHash = TestBodyHash
              -590681868797176966
              +590680769285548757,
            thSlotNo = SlotNo -0 +1,
            thBlockNo = BlockNo -1 +2,
            thChainLength = ChainLength
              -2
              +3,
            thIsEBB = RegularBlock},
          testBody = TestBody {
            tbForkNo = -2 +3,
            tbIsValid = True}},
      nextFollowerId = 0,
      chainFollowers = ...},
    currentLedger = ExtLedgerState {
      ledgerState = TestLedger {
        lastAppliedPoint = Point
          (At
            Block {
              blockPointSlot = SlotNo -0 +1,
              blockPointHash = TestHeaderHash
                -9122512639099877219
                +`-7915818155603348863`}),
        lastAppliedHash = BlockHash
          (TestHeaderHash
            -9122512639099877219
            +`-7915818155603348863`)},
      headerState = HeaderState {
        headerStateTip = At
          AnnTip {
            annTipSlotNo = SlotNo -0 +1,
            annTipBlockNo = BlockNo -1 +2,
            annTipInfo = TipInfoIsEBB
              (TestHeaderHash
                -9122512639099877219
                +`-7915818155603348863`)
              IsNotEBB},
        headerStateChainDep = `()`}},
    valid = Set.fromList
      [
        +TestHeaderHash
          `-7915818155603348863`,
        TestHeaderHash
          2173152789646624309,
        TestHeaderHash
          9122512639099877219],
    currentSlot = SlotNo -0 +1,
    maxClockSkew = 100000,
    isOpen = True,
    initLedger = ...},
  modelConfig = Opaque,
  knownIters = ...}

   == At
  { unAt =
      AddBlock
        TestBlock
          { testHeader =
              TestHeader
                { thHash = TestHeaderHash (-1393368485677491682)
                , thPrevHash = BlockHash (TestHeaderHash (-7915818155603348863))
                , thBodyHash = TestBodyHash 590682968308805178
                , thSlotNo = SlotNo 3
                , thBlockNo = BlockNo 3
                , thChainLength = ChainLength 4
                , thIsEBB = RegularBlock
                }
          , testBody = TestBody { tbForkNo = 1 , tbIsValid = False }
          }
  } ==> At
  { unAt =
      Resp
        { getResp =
            Right
              (Point
                 (At
                    Block
                      { blockPointSlot = SlotNo 1
                      , blockPointHash = TestHeaderHash (-7915818155603348863)
                      }))
        }
  } [ 0 ]

Model {
  dbModel = Model {
    volatileDbBlocks = Map.fromList
      [
        _×_
          (TestHeaderHash
            `-7915818155603348863`)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                `-7915818155603348863`,
              thPrevHash = BlockHash
                (TestHeaderHash
                  9122512639099877219),
              thBodyHash = TestBodyHash
                590680769285548757,
              thSlotNo = SlotNo 1,
              thBlockNo = BlockNo 2,
              thChainLength = ChainLength 3,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 3,
              tbIsValid = True}},
        +_×_
          (TestHeaderHash
            `-1393368485677491682`)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                `-1393368485677491682`,
              thPrevHash = BlockHash
                (TestHeaderHash
                  `-7915818155603348863`),
              thBodyHash = TestBodyHash
                590682968308805178,
              thSlotNo = SlotNo 3,
              thBlockNo = BlockNo 3,
              thChainLength = ChainLength 4,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 1,
              tbIsValid = False}},
        _×_
          (TestHeaderHash
            2173152789646624309)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                2173152789646624309,
              thPrevHash = GenesisHash,
              thBodyHash = TestBodyHash
                590680769285548757,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 0,
              thChainLength = ChainLength 1,
              thIsEBB = EBB (EpochNo 0)},
            testBody = TestBody {
              tbForkNo = 3,
              tbIsValid = True}},
        _×_
          (TestHeaderHash
            9122512639099877219)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                9122512639099877219,
              thPrevHash = BlockHash
                (TestHeaderHash
                  2173152789646624309),
              thBodyHash = TestBodyHash
                590681868797176966,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 1,
              thChainLength = ChainLength 2,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 2,
              tbIsValid = True}}],
    immutableDbChain = Genesis,
    invalid = Map.fromList
      [
        +_×_
          (TestHeaderHash
            `-1393368485677491682`)
          (_×_
            (ValidationError
              (ExtValidationErrorLedger
                InvalidBlock))
            (SlotNo 3))],
    currentSlot = SlotNo -1 +3,
    maxClockSkew = 100000,
    isOpen = True,
    cps = ...},
  modelConfig = Opaque,
  knownIters = ...}

   == At
  { unAt =
      AddBlock
        TestBlock
          { testHeader =
              TestHeader
                { thHash = TestHeaderHash (-2651776708921877814)
                , thPrevHash = BlockHash (TestHeaderHash 2173152789646624309)
                , thBodyHash = TestBodyHash 590682968308805179
                , thSlotNo = SlotNo 0
                , thBlockNo = BlockNo 1
                , thChainLength = ChainLength 2
                , thIsEBB = RegularBlock
                }
          , testBody = TestBody { tbForkNo = 1 , tbIsValid = True }
          }
  } ==> At
  { unAt =
      Resp
        { getResp =
            Right
              (Point
                 (At
                    Block
                      { blockPointSlot = SlotNo 1
                      , blockPointHash = TestHeaderHash (-7915818155603348863)
                      }))
        }
  } [ 0 ]

Model {
  dbModel = Model {
    volatileDbBlocks = Map.fromList
      [
        _×_
          (TestHeaderHash
            `-7915818155603348863`)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                `-7915818155603348863`,
              thPrevHash = BlockHash
                (TestHeaderHash
                  9122512639099877219),
              thBodyHash = TestBodyHash
                590680769285548757,
              thSlotNo = SlotNo 1,
              thBlockNo = BlockNo 2,
              thChainLength = ChainLength 3,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 3,
              tbIsValid = True}},
        +_×_
          (TestHeaderHash
            `-2651776708921877814`)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                `-2651776708921877814`,
              thPrevHash = BlockHash
                (TestHeaderHash
                  2173152789646624309),
              thBodyHash = TestBodyHash
                590682968308805179,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 1,
              thChainLength = ChainLength 2,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 1,
              tbIsValid = True}},
        _×_
          (TestHeaderHash
            `-1393368485677491682`)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                `-1393368485677491682`,
              thPrevHash = BlockHash
                (TestHeaderHash
                  `-7915818155603348863`),
              thBodyHash = TestBodyHash
                590682968308805178,
              thSlotNo = SlotNo 3,
              thBlockNo = BlockNo 3,
              thChainLength = ChainLength 4,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 1,
              tbIsValid = False}},
        _×_
          (TestHeaderHash
            2173152789646624309)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                2173152789646624309,
              thPrevHash = GenesisHash,
              thBodyHash = TestBodyHash
                590680769285548757,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 0,
              thChainLength = ChainLength 1,
              thIsEBB = EBB (EpochNo 0)},
            testBody = TestBody {
              tbForkNo = 3,
              tbIsValid = True}},
        _×_
          (TestHeaderHash
            9122512639099877219)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                9122512639099877219,
              thPrevHash = BlockHash
                (TestHeaderHash
                  2173152789646624309),
              thBodyHash = TestBodyHash
                590681868797176966,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 1,
              thChainLength = ChainLength 2,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 2,
              tbIsValid = True}}],
    immutableDbChain = Genesis,
    maxClockSkew = 100000,
    isOpen = True,
    cps = ...},
  modelConfig = Opaque,
  knownIters = ...}

   == At
  { unAt =
      AddBlock
        TestBlock
          { testHeader =
              TestHeader
                { thHash = TestHeaderHash 128720366063791916
                , thPrevHash = BlockHash (TestHeaderHash (-2651776708921877814))
                , thBodyHash = TestBodyHash 590682968308805178
                , thSlotNo = SlotNo 2
                , thBlockNo = BlockNo 2
                , thChainLength = ChainLength 3
                , thIsEBB = RegularBlock
                }
          , testBody = TestBody { tbForkNo = 1 , tbIsValid = False }
          }
  } ==> At
  { unAt =
      Resp
        { getResp =
            Right
              (Point
                 (At
                    Block
                      { blockPointSlot = SlotNo 1
                      , blockPointHash = TestHeaderHash (-7915818155603348863)
                      }))
        }
  } [ 0 ]

Model {
  dbModel = Model {
    volatileDbBlocks = Map.fromList
      [
        _×_
          (TestHeaderHash
            `-7915818155603348863`)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                `-7915818155603348863`,
              thPrevHash = BlockHash
                (TestHeaderHash
                  9122512639099877219),
              thBodyHash = TestBodyHash
                590680769285548757,
              thSlotNo = SlotNo 1,
              thBlockNo = BlockNo 2,
              thChainLength = ChainLength 3,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 3,
              tbIsValid = True}},
        _×_
          (TestHeaderHash
            `-2651776708921877814`)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                `-2651776708921877814`,
              thPrevHash = BlockHash
                (TestHeaderHash
                  2173152789646624309),
              thBodyHash = TestBodyHash
                590682968308805179,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 1,
              thChainLength = ChainLength 2,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 1,
              tbIsValid = True}},
        _×_
          (TestHeaderHash
            `-1393368485677491682`)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                `-1393368485677491682`,
              thPrevHash = BlockHash
                (TestHeaderHash
                  `-7915818155603348863`),
              thBodyHash = TestBodyHash
                590682968308805178,
              thSlotNo = SlotNo 3,
              thBlockNo = BlockNo 3,
              thChainLength = ChainLength 4,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 1,
              tbIsValid = False}},
        +_×_
          (TestHeaderHash
            128720366063791916)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                128720366063791916,
              thPrevHash = BlockHash
                (TestHeaderHash
                  `-2651776708921877814`),
              thBodyHash = TestBodyHash
                590682968308805178,
              thSlotNo = SlotNo 2,
              thBlockNo = BlockNo 2,
              thChainLength = ChainLength 3,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 1,
              tbIsValid = False}},
        _×_
          (TestHeaderHash
            2173152789646624309)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                2173152789646624309,
              thPrevHash = GenesisHash,
              thBodyHash = TestBodyHash
                590680769285548757,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 0,
              thChainLength = ChainLength 1,
              thIsEBB = EBB (EpochNo 0)},
            testBody = TestBody {
              tbForkNo = 3,
              tbIsValid = True}},
        _×_
          (TestHeaderHash
            9122512639099877219)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                9122512639099877219,
              thPrevHash = BlockHash
                (TestHeaderHash
                  2173152789646624309),
              thBodyHash = TestBodyHash
                590681868797176966,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 1,
              thChainLength = ChainLength 2,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 2,
              tbIsValid = True}}],
    immutableDbChain = Genesis,
    invalid = Map.fromList
      [
        _×_
          (TestHeaderHash
            `-1393368485677491682`)
          (_×_
            (ValidationError
              (ExtValidationErrorLedger
                InvalidBlock))
            (SlotNo 3)),
        +_×_
          (TestHeaderHash
            128720366063791916)
          (_×_
            (ValidationError
              (ExtValidationErrorLedger
                InvalidBlock))
            (SlotNo 2))],
    maxClockSkew = 100000,
    isOpen = True,
    cps = ...},
  modelConfig = Opaque,
  knownIters = ...}

   == At
  { unAt =
      GetIsValid
        (RealPoint (SlotNo 0) (TestHeaderHash (-2651776708921877814)))
  } ==> At
  { unAt =
      Resp
        { getResp =
            Right (IsValid IsValidResult { real = True , isValid = Just True })
        }
  } [ 0 ]

Model {
  dbModel = Model {
    volatileDbBlocks = Map.fromList
      [
        _×_
          (TestHeaderHash
            `-7915818155603348863`)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                `-7915818155603348863`,
              thPrevHash = BlockHash
                (TestHeaderHash
                  9122512639099877219),
              thBodyHash = TestBodyHash
                590680769285548757,
              thSlotNo = SlotNo 1,
              thBlockNo = BlockNo 2,
              thChainLength = ChainLength 3,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 3,
              tbIsValid = True}},
        _×_
          (TestHeaderHash
            `-2651776708921877814`)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                `-2651776708921877814`,
              thPrevHash = BlockHash
                (TestHeaderHash
                  2173152789646624309),
              thBodyHash = TestBodyHash
                590682968308805179,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 1,
              thChainLength = ChainLength 2,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 1,
              tbIsValid = True}},
        _×_
          (TestHeaderHash
            `-1393368485677491682`)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                `-1393368485677491682`,
              thPrevHash = BlockHash
                (TestHeaderHash
                  `-7915818155603348863`),
              thBodyHash = TestBodyHash
                590682968308805178,
              thSlotNo = SlotNo 3,
              thBlockNo = BlockNo 3,
              thChainLength = ChainLength 4,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 1,
              tbIsValid = False}},
        _×_
          (TestHeaderHash
            128720366063791916)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                128720366063791916,
              thPrevHash = BlockHash
                (TestHeaderHash
                  `-2651776708921877814`),
              thBodyHash = TestBodyHash
                590682968308805178,
              thSlotNo = SlotNo 2,
              thBlockNo = BlockNo 2,
              thChainLength = ChainLength 3,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 1,
              tbIsValid = False}},
        _×_
          (TestHeaderHash
            2173152789646624309)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                2173152789646624309,
              thPrevHash = GenesisHash,
              thBodyHash = TestBodyHash
                590680769285548757,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 0,
              thChainLength = ChainLength 1,
              thIsEBB = EBB (EpochNo 0)},
            testBody = TestBody {
              tbForkNo = 3,
              tbIsValid = True}},
        _×_
          (TestHeaderHash
            9122512639099877219)
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                9122512639099877219,
              thPrevHash = BlockHash
                (TestHeaderHash
                  2173152789646624309),
              thBodyHash = TestBodyHash
                590681868797176966,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 1,
              thChainLength = ChainLength 2,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 2,
              tbIsValid = True}}],
    immutableDbChain = Genesis,
    cps = ChainProducerState {
      chainState = :>
        (:>
          (:>
            Genesis
            TestBlock {
              testHeader = TestHeader {
                thHash = TestHeaderHash
                  2173152789646624309,
                thPrevHash = GenesisHash,
                thBodyHash = TestBodyHash
                  590680769285548757,
                thSlotNo = SlotNo 0,
                thBlockNo = BlockNo 0,
                thChainLength = ChainLength 1,
                thIsEBB = EBB (EpochNo 0)},
              testBody = TestBody {
                tbForkNo = 3,
                tbIsValid = True}})
          TestBlock {
            testHeader = TestHeader {
              thHash = TestHeaderHash
                9122512639099877219,
              thPrevHash = BlockHash
                (TestHeaderHash
                  2173152789646624309),
              thBodyHash = TestBodyHash
                590681868797176966,
              thSlotNo = SlotNo 0,
              thBlockNo = BlockNo 1,
              thChainLength = ChainLength 2,
              thIsEBB = RegularBlock},
            testBody = TestBody {
              tbForkNo = 2,
              tbIsValid = True}})
        TestBlock {
          testHeader = TestHeader {
            thHash = TestHeaderHash
              `-7915818155603348863`,
            thPrevHash = BlockHash
              (TestHeaderHash
                9122512639099877219),
            thBodyHash = TestBodyHash
              590680769285548757,
            thSlotNo = SlotNo 1,
            thBlockNo = BlockNo 2,
            thChainLength = ChainLength 3,
            thIsEBB = RegularBlock},
          testBody = TestBody {
            tbForkNo = 3,
            tbIsValid = True}},
      chainFollowers = Map.fromList
        [],
      nextFollowerId = 0},
    currentLedger = ExtLedgerState {
      ledgerState = TestLedger {
        lastAppliedPoint = Point
          (At
            Block {
              blockPointSlot = SlotNo 1,
              blockPointHash = TestHeaderHash
                `-7915818155603348863`}),
        lastAppliedHash = BlockHash
          (TestHeaderHash
            `-7915818155603348863`)},
      headerState = HeaderState {
        headerStateTip = At
          AnnTip {
            annTipSlotNo = SlotNo 1,
            annTipBlockNo = BlockNo 2,
            annTipInfo = TipInfoIsEBB
              (TestHeaderHash
                `-7915818155603348863`)
              IsNotEBB},
        headerStateChainDep = `()`}},
    initLedger = ExtLedgerState {
      ledgerState = TestLedger {
        lastAppliedPoint = Point Origin,
        lastAppliedHash = GenesisHash},
      headerState = HeaderState {
        headerStateTip = Origin,
        headerStateChainDep = `()`}},
    iterators = Map.fromList [],
    valid = Set.fromList
      [
        TestHeaderHash
          `-7915818155603348863`,
        TestHeaderHash
          2173152789646624309,
        TestHeaderHash
          9122512639099877219],
    invalid = Map.fromList
      [
        _×_
          (TestHeaderHash
            `-1393368485677491682`)
          (_×_
            (ValidationError
              (ExtValidationErrorLedger
                InvalidBlock))
            (SlotNo 3)),
        _×_
          (TestHeaderHash
            128720366063791916)
          (_×_
            (ValidationError
              (ExtValidationErrorLedger
                InvalidBlock))
            (SlotNo 2))],
    currentSlot = SlotNo 3,
    maxClockSkew = 100000,
    isOpen = True},
  knownIters = RefEnv [],
  knownFollowers = RefEnv [],
  modelConfig = Opaque}

FAIL (99.21s)
          *** Failed! Falsified (after 50558 tests and 15 shrinks):
          MaxClockSkew 100000
          SmallChunkInfo (UniformChunkSize (ChunkSize {chunkCanContainEBB = True, numRegularBlocks = 14}))
          Commands
            { unCommands =
                [ Command
                    At
                      { unAt =
                          AddBlock
                            TestBlock
                              { testHeader =
                                  TestHeader
                                    { thHash = TestHeaderHash 2173152789646624309
                                    , thPrevHash = GenesisHash
                                    , thBodyHash = TestBodyHash 590680769285548757
                                    , thSlotNo = SlotNo 0
                                    , thBlockNo = BlockNo 0
                                    , thChainLength = ChainLength 1
                                    , thIsEBB = EBB (EpochNo 0)
                                    }
                              , testBody = TestBody { tbForkNo = 3 , tbIsValid = True }
                              }
                      }
                    At
                      { unAt =
                          Resp
                            { getResp =
                                Right
                                  (Point
                                     (At
                                        Block
                                          { blockPointSlot = SlotNo 0
                                          , blockPointHash = TestHeaderHash 2173152789646624309
                                          }))
                            }
                      }
                    []
                , Command
                    At
                      { unAt =
                          AddBlock
                            TestBlock
                              { testHeader =
                                  TestHeader
                                    { thHash = TestHeaderHash 9122512639099877219
                                    , thPrevHash = BlockHash (TestHeaderHash 2173152789646624309)
                                    , thBodyHash = TestBodyHash 590681868797176966
                                    , thSlotNo = SlotNo 0
                                    , thBlockNo = BlockNo 1
                                    , thChainLength = ChainLength 2
                                    , thIsEBB = RegularBlock
                                    }
                              , testBody = TestBody { tbForkNo = 2 , tbIsValid = True }
                              }
                      }
                    At
                      { unAt =
                          Resp
                            { getResp =
                                Right
                                  (Point
                                     (At
                                        Block
                                          { blockPointSlot = SlotNo 0
                                          , blockPointHash = TestHeaderHash 9122512639099877219
                                          }))
                            }
                      }
                    []
                , Command
                    At
                      { unAt =
                          AddBlock
                            TestBlock
                              { testHeader =
                                  TestHeader
                                    { thHash = TestHeaderHash (-7915818155603348863)
                                    , thPrevHash = BlockHash (TestHeaderHash 9122512639099877219)
                                    , thBodyHash = TestBodyHash 590680769285548757
                                    , thSlotNo = SlotNo 1
                                    , thBlockNo = BlockNo 2
                                    , thChainLength = ChainLength 3
                                    , thIsEBB = RegularBlock
                                    }
                              , testBody = TestBody { tbForkNo = 3 , tbIsValid = True }
                              }
                      }
                    At
                      { unAt =
                          Resp
                            { getResp =
                                Right
                                  (Point
                                     (At
                                        Block
                                          { blockPointSlot = SlotNo 1
                                          , blockPointHash = TestHeaderHash (-7915818155603348863)
                                          }))
                            }
                      }
                    []
                , Command
                    At
                      { unAt =
                          AddBlock
                            TestBlock
                              { testHeader =
                                  TestHeader
                                    { thHash = TestHeaderHash (-1393368485677491682)
                                    , thPrevHash = BlockHash (TestHeaderHash (-7915818155603348863))
                                    , thBodyHash = TestBodyHash 590682968308805178
                                    , thSlotNo = SlotNo 3
                                    , thBlockNo = BlockNo 3
                                    , thChainLength = ChainLength 4
                                    , thIsEBB = RegularBlock
                                    }
                              , testBody = TestBody { tbForkNo = 1 , tbIsValid = False }
                              }
                      }
                    At
                      { unAt =
                          Resp
                            { getResp =
                                Right
                                  (Point
                                     (At
                                        Block
                                          { blockPointSlot = SlotNo 1
                                          , blockPointHash = TestHeaderHash (-7915818155603348863)
                                          }))
                            }
                      }
                    []
                , Command
                    At
                      { unAt =
                          AddBlock
                            TestBlock
                              { testHeader =
                                  TestHeader
                                    { thHash = TestHeaderHash (-2651776708921877814)
                                    , thPrevHash = BlockHash (TestHeaderHash 2173152789646624309)
                                    , thBodyHash = TestBodyHash 590682968308805179
                                    , thSlotNo = SlotNo 0
                                    , thBlockNo = BlockNo 1
                                    , thChainLength = ChainLength 2
                                    , thIsEBB = RegularBlock
                                    }
                              , testBody = TestBody { tbForkNo = 1 , tbIsValid = True }
                              }
                      }
                    At
                      { unAt =
                          Resp
                            { getResp =
                                Right
                                  (Point
                                     (At
                                        Block
                                          { blockPointSlot = SlotNo 1
                                          , blockPointHash = TestHeaderHash (-7915818155603348863)
                                          }))
                            }
                      }
                    []
                , Command
                    At
                      { unAt =
                          AddBlock
                            TestBlock
                              { testHeader =
                                  TestHeader
                                    { thHash = TestHeaderHash 128720366063791916
                                    , thPrevHash = BlockHash (TestHeaderHash (-2651776708921877814))
                                    , thBodyHash = TestBodyHash 590682968308805178
                                    , thSlotNo = SlotNo 2
                                    , thBlockNo = BlockNo 2
                                    , thChainLength = ChainLength 3
                                    , thIsEBB = RegularBlock
                                    }
                              , testBody = TestBody { tbForkNo = 1 , tbIsValid = False }
                              }
                      }
                    At
                      { unAt =
                          Resp
                            { getResp =
                                Right
                                  (Point
                                     (At
                                        Block
                                          { blockPointSlot = SlotNo 1
                                          , blockPointHash = TestHeaderHash (-7915818155603348863)
                                          }))
                            }
                      }
                    []
                , Command
                    At
                      { unAt =
                          GetIsValid
                            (RealPoint (SlotNo 0) (TestHeaderHash (-2651776708921877814)))
                      }
                    At
                      { unAt =
                          Resp
                            { getResp =
                                Right (IsValid IsValidResult { real = False , isValid = Nothing })
                            }
                      }
                    []
                ]
            }
          Model chain: Genesis :> TestBlock {testHeader = TestHeader {thHash = TestHeaderHash 2173152789646624309, thPrevHash = GenesisHash, thBodyHash = TestBodyHash 590680769285548757, thSlotNo = SlotNo 0, thBlockNo = BlockNo 0, thChainLength = ChainLength 1, thIsEBB = EBB (EpochNo 0)}, testBody = TestBody {tbForkNo = 3, tbIsValid = True}} :> TestBlock {testHeader = TestHeader {thHash = TestHeaderHash 9122512639099877219, thPrevHash = BlockHash (TestHeaderHash 2173152789646624309), thBodyHash = TestBodyHash 590681868797176966, thSlotNo = SlotNo 0, thBlockNo = BlockNo 1, thChainLength = ChainLength 2, thIsEBB = RegularBlock}, testBody = TestBody {tbForkNo = 2, tbIsValid = True}} :> TestBlock {testHeader = TestHeader {thHash = TestHeaderHash (-7915818155603348863), thPrevHash = BlockHash (TestHeaderHash 9122512639099877219), thBodyHash = TestBodyHash 590680769285548757, thSlotNo = SlotNo 1, thBlockNo = BlockNo 2, thChainLength = ChainLength 3, thIsEBB = RegularBlock}, testBody = TestBody {tbForkNo = 3, tbIsValid = True}}
          TraceEvents: TraceOpenEvent StartedOpeningDB
          TraceOpenEvent StartedOpeningImmutableDB
          TraceImmutableDBEvent NoValidLastLocation
          TraceOpenEvent (OpenedImmutableDB Origin 0)
          TraceOpenEvent StartedOpeningVolatileDB
          TraceOpenEvent OpenedVolatileDB
          TraceOpenEvent StartedOpeningLgrDB
          TraceLedgerReplayEvent (ReplayFromGenesis (ReplayGoal Origin))
          TraceOpenEvent OpenedLgrDB
          TraceInitChainSelEvent StartedInitChainSelection
          TraceInitChainSelEvent InitalChainSelected
          TraceOpenEvent (OpenedDB Origin Origin)
          TraceAddBlockEvent (AddedBlockToQueue (RealPoint (SlotNo 0) (TestHeaderHash 2173152789646624309)) 1)
          TraceAddBlockEvent (AddedBlockToVolatileDB (RealPoint (SlotNo 0) (TestHeaderHash 2173152789646624309)) (BlockNo 0) IsEBB)
          TraceAddBlockEvent (TryAddToCurrentChain (RealPoint (SlotNo 0) (TestHeaderHash 2173152789646624309)))
          TraceAddBlockEvent (AddBlockValidation (UpdateLedgerDbTraceEvent (StartedPushingBlockToTheLedgerDb (PushStart {unPushStart = RealPoint (SlotNo 0) (TestHeaderHash 2173152789646624309)}) (PushGoal {unPushGoal = RealPoint (SlotNo 0) (TestHeaderHash 2173152789646624309)}) (Pushing {unPushing = RealPoint (SlotNo 0) (TestHeaderHash 2173152789646624309)}))))
          TraceAddBlockEvent (AddBlockValidation (ValidCandidate (AnchoredSeq {anchor = AnchorGenesis, unanchorSeq = SFT {fromStrict = fromList [MeasuredWith {unMeasuredWith = TestHeader {thHash = TestHeaderHash 2173152789646624309, thPrevHash = GenesisHash, thBodyHash = TestBodyHash 590680769285548757, thSlotNo = SlotNo 0, thBlockNo = BlockNo 0, thChainLength = ChainLength 1, thIsEBB = EBB (EpochNo 0)}}]}})))
          TraceAddBlockEvent (AddedToCurrentChain [] (NewTipInfo {newTipPoint = RealPoint (SlotNo 0) (TestHeaderHash 2173152789646624309), newTipEpoch = EpochNo 0, newTipSlotInEpoch = 0, newTipTrigger = RealPoint (SlotNo 0) (TestHeaderHash 2173152789646624309)}) (AnchoredSeq {anchor = AnchorGenesis, unanchorSeq = SFT {fromStrict = fromList []}}) (AnchoredSeq {anchor = AnchorGenesis, unanchorSeq = SFT {fromStrict = fromList [MeasuredWith {unMeasuredWith = TestHeader {thHash = TestHeaderHash 2173152789646624309, thPrevHash = GenesisHash, thBodyHash = TestBodyHash 590680769285548757, thSlotNo = SlotNo 0, thBlockNo = BlockNo 0, thChainLength = ChainLength 1, thIsEBB = EBB (EpochNo 0)}}]}}))
          TraceAddBlockEvent (AddedBlockToQueue (RealPoint (SlotNo 0) (TestHeaderHash 9122512639099877219)) 1)
          TraceAddBlockEvent (AddedBlockToVolatileDB (RealPoint (SlotNo 0) (TestHeaderHash 9122512639099877219)) (BlockNo 1) IsNotEBB)
          TraceAddBlockEvent (TryAddToCurrentChain (RealPoint (SlotNo 0) (TestHeaderHash 9122512639099877219)))
          TraceAddBlockEvent (AddBlockValidation (UpdateLedgerDbTraceEvent (StartedPushingBlockToTheLedgerDb (PushStart {unPushStart = RealPoint (SlotNo 0) (TestHeaderHash 9122512639099877219)}) (PushGoal {unPushGoal = RealPoint (SlotNo 0) (TestHeaderHash 9122512639099877219)}) (Pushing {unPushing = RealPoint (SlotNo 0) (TestHeaderHash 9122512639099877219)}))))
          TraceAddBlockEvent (AddBlockValidation (ValidCandidate (AnchoredSeq {anchor = Anchor (SlotNo 0) (TestHeaderHash 2173152789646624309) (BlockNo 0), unanchorSeq = SFT {fromStrict = fromList [MeasuredWith {unMeasuredWith = TestHeader {thHash = TestHeaderHash 9122512639099877219, thPrevHash = BlockHash (TestHeaderHash 2173152789646624309), thBodyHash = TestBodyHash 590681868797176966, thSlotNo = SlotNo 0, thBlockNo = BlockNo 1, thChainLength = ChainLength 2, thIsEBB = RegularBlock}}]}})))
          TraceAddBlockEvent (AddedToCurrentChain [] (NewTipInfo {newTipPoint = RealPoint (SlotNo 0) (TestHeaderHash 9122512639099877219), newTipEpoch = EpochNo 0, newTipSlotInEpoch = 0, newTipTrigger = RealPoint (SlotNo 0) (TestHeaderHash 9122512639099877219)}) (AnchoredSeq {anchor = AnchorGenesis, unanchorSeq = SFT {fromStrict = fromList [MeasuredWith {unMeasuredWith = TestHeader {thHash = TestHeaderHash 2173152789646624309, thPrevHash = GenesisHash, thBodyHash = TestBodyHash 590680769285548757, thSlotNo = SlotNo 0, thBlockNo = BlockNo 0, thChainLength = ChainLength 1, thIsEBB = EBB (EpochNo 0)}}]}}) (AnchoredSeq {anchor = AnchorGenesis, unanchorSeq = SFT {fromStrict = fromList [MeasuredWith {unMeasuredWith = TestHeader {thHash = TestHeaderHash 2173152789646624309, thPrevHash = GenesisHash, thBodyHash = TestBodyHash 590680769285548757, thSlotNo = SlotNo 0, thBlockNo = BlockNo 0, thChainLength = ChainLength 1, thIsEBB = EBB (EpochNo 0)}},MeasuredWith {unMeasuredWith = TestHeader {thHash = TestHeaderHash 9122512639099877219, thPrevHash = BlockHash (TestHeaderHash 2173152789646624309), thBodyHash = TestBodyHash 590681868797176966, thSlotNo = SlotNo 0, thBlockNo = BlockNo 1, thChainLength = ChainLength 2, thIsEBB = RegularBlock}}]}}))
          TraceAddBlockEvent (AddedBlockToQueue (RealPoint (SlotNo 1) (TestHeaderHash (-7915818155603348863))) 1)
          TraceAddBlockEvent (AddedBlockToVolatileDB (RealPoint (SlotNo 1) (TestHeaderHash (-7915818155603348863))) (BlockNo 2) IsNotEBB)
          TraceAddBlockEvent (TryAddToCurrentChain (RealPoint (SlotNo 1) (TestHeaderHash (-7915818155603348863))))
          TraceAddBlockEvent (AddBlockValidation (UpdateLedgerDbTraceEvent (StartedPushingBlockToTheLedgerDb (PushStart {unPushStart = RealPoint (SlotNo 1) (TestHeaderHash (-7915818155603348863))}) (PushGoal {unPushGoal = RealPoint (SlotNo 1) (TestHeaderHash (-7915818155603348863))}) (Pushing {unPushing = RealPoint (SlotNo 1) (TestHeaderHash (-7915818155603348863))}))))
          TraceAddBlockEvent (AddBlockValidation (ValidCandidate (AnchoredSeq {anchor = Anchor (SlotNo 0) (TestHeaderHash 9122512639099877219) (BlockNo 1), unanchorSeq = SFT {fromStrict = fromList [MeasuredWith {unMeasuredWith = TestHeader {thHash = TestHeaderHash (-7915818155603348863), thPrevHash = BlockHash (TestHeaderHash 9122512639099877219), thBodyHash = TestBodyHash 590680769285548757, thSlotNo = SlotNo 1, thBlockNo = BlockNo 2, thChainLength = ChainLength 3, thIsEBB = RegularBlock}}]}})))
          TraceAddBlockEvent (AddedToCurrentChain [] (NewTipInfo {newTipPoint = RealPoint (SlotNo 1) (TestHeaderHash (-7915818155603348863)), newTipEpoch = EpochNo 0, newTipSlotInEpoch = 1, newTipTrigger = RealPoint (SlotNo 1) (TestHeaderHash (-7915818155603348863))}) (AnchoredSeq {anchor = AnchorGenesis, unanchorSeq = SFT {fromStrict = fromList [MeasuredWith {unMeasuredWith = TestHeader {thHash = TestHeaderHash 2173152789646624309, thPrevHash = GenesisHash, thBodyHash = TestBodyHash 590680769285548757, thSlotNo = SlotNo 0, thBlockNo = BlockNo 0, thChainLength = ChainLength 1, thIsEBB = EBB (EpochNo 0)}},MeasuredWith {unMeasuredWith = TestHeader {thHash = TestHeaderHash 9122512639099877219, thPrevHash = BlockHash (TestHeaderHash 2173152789646624309), thBodyHash = TestBodyHash 590681868797176966, thSlotNo = SlotNo 0, thBlockNo = BlockNo 1, thChainLength = ChainLength 2, thIsEBB = RegularBlock}}]}}) (AnchoredSeq {anchor = AnchorGenesis, unanchorSeq = SFT {fromStrict = fromList [MeasuredWith {unMeasuredWith = TestHeader {thHash = TestHeaderHash 2173152789646624309, thPrevHash = GenesisHash, thBodyHash = TestBodyHash 590680769285548757, thSlotNo = SlotNo 0, thBlockNo = BlockNo 0, thChainLength = ChainLength 1, thIsEBB = EBB (EpochNo 0)}},MeasuredWith {unMeasuredWith = TestHeader {thHash = TestHeaderHash 9122512639099877219, thPrevHash = BlockHash (TestHeaderHash 2173152789646624309), thBodyHash = TestBodyHash 590681868797176966, thSlotNo = SlotNo 0, thBlockNo = BlockNo 1, thChainLength = ChainLength 2, thIsEBB = RegularBlock}},MeasuredWith {unMeasuredWith = TestHeader {thHash = TestHeaderHash (-7915818155603348863), thPrevHash = BlockHash (TestHeaderHash 9122512639099877219), thBodyHash = TestBodyHash 590680769285548757, thSlotNo = SlotNo 1, thBlockNo = BlockNo 2, thChainLength = ChainLength 3, thIsEBB = RegularBlock}}]}}))
          TraceAddBlockEvent (AddedBlockToQueue (RealPoint (SlotNo 3) (TestHeaderHash (-1393368485677491682))) 1)
          TraceAddBlockEvent (AddedBlockToVolatileDB (RealPoint (SlotNo 3) (TestHeaderHash (-1393368485677491682))) (BlockNo 3) IsNotEBB)
          TraceAddBlockEvent (TryAddToCurrentChain (RealPoint (SlotNo 3) (TestHeaderHash (-1393368485677491682))))
          TraceAddBlockEvent (AddBlockValidation (UpdateLedgerDbTraceEvent (StartedPushingBlockToTheLedgerDb (PushStart {unPushStart = RealPoint (SlotNo 3) (TestHeaderHash (-1393368485677491682))}) (PushGoal {unPushGoal = RealPoint (SlotNo 3) (TestHeaderHash (-1393368485677491682))}) (Pushing {unPushing = RealPoint (SlotNo 3) (TestHeaderHash (-1393368485677491682))}))))
          TraceAddBlockEvent (AddBlockValidation (InvalidBlock (ExtValidationErrorLedger InvalidBlock) (RealPoint (SlotNo 3) (TestHeaderHash (-1393368485677491682)))))
          TraceAddBlockEvent (AddBlockValidation (ValidCandidate (AnchoredSeq {anchor = Anchor (SlotNo 1) (TestHeaderHash (-7915818155603348863)) (BlockNo 2), unanchorSeq = SFT {fromStrict = fromList []}})))
          TraceAddBlockEvent (AddedBlockToQueue (RealPoint (SlotNo 0) (TestHeaderHash (-2651776708921877814))) 1)
          TraceAddBlockEvent (AddedBlockToVolatileDB (RealPoint (SlotNo 0) (TestHeaderHash (-2651776708921877814))) (BlockNo 1) IsNotEBB)
          TraceAddBlockEvent (TrySwitchToAFork (RealPoint (SlotNo 0) (TestHeaderHash (-2651776708921877814))) (ChainDiff {getRollback = 2, getSuffix = AnchoredSeq {anchor = Anchor (SlotNo 0) (TestHeaderHash 2173152789646624309) (BlockNo 0), unanchorSeq = SFT {fromStrict = fromList [MeasuredWith {unMeasuredWith = HeaderFields {headerFieldSlot = SlotNo 0, headerFieldBlockNo = BlockNo 1, headerFieldHash = TestHeaderHash (-2651776708921877814)}}]}}}))
          TraceAddBlockEvent (AddedBlockToQueue (RealPoint (SlotNo 2) (TestHeaderHash 128720366063791916)) 1)
          TraceAddBlockEvent (AddedBlockToVolatileDB (RealPoint (SlotNo 2) (TestHeaderHash 128720366063791916)) (BlockNo 2) IsNotEBB)
          TraceAddBlockEvent (TrySwitchToAFork (RealPoint (SlotNo 2) (TestHeaderHash 128720366063791916)) (ChainDiff {getRollback = 2, getSuffix = AnchoredSeq {anchor = Anchor (SlotNo 0) (TestHeaderHash 2173152789646624309) (BlockNo 0), unanchorSeq = SFT {fromStrict = fromList [MeasuredWith {unMeasuredWith = HeaderFields {headerFieldSlot = SlotNo 0, headerFieldBlockNo = BlockNo 1, headerFieldHash = TestHeaderHash (-2651776708921877814)}},MeasuredWith {unMeasuredWith = HeaderFields {headerFieldSlot = SlotNo 2, headerFieldBlockNo = BlockNo 2, headerFieldHash = TestHeaderHash 128720366063791916}}]}}}))
          TraceAddBlockEvent (AddBlockValidation (UpdateLedgerDbTraceEvent (StartedPushingBlockToTheLedgerDb (PushStart {unPushStart = RealPoint (SlotNo 0) (TestHeaderHash (-2651776708921877814))}) (PushGoal {unPushGoal = RealPoint (SlotNo 2) (TestHeaderHash 128720366063791916)}) (Pushing {unPushing = RealPoint (SlotNo 0) (TestHeaderHash (-2651776708921877814))}))))
          TraceAddBlockEvent (AddBlockValidation (UpdateLedgerDbTraceEvent (StartedPushingBlockToTheLedgerDb (PushStart {unPushStart = RealPoint (SlotNo 0) (TestHeaderHash (-2651776708921877814))}) (PushGoal {unPushGoal = RealPoint (SlotNo 2) (TestHeaderHash 128720366063791916)}) (Pushing {unPushing = RealPoint (SlotNo 2) (TestHeaderHash 128720366063791916)}))))
          TraceAddBlockEvent (AddBlockValidation (InvalidBlock (ExtValidationErrorLedger InvalidBlock) (RealPoint (SlotNo 2) (TestHeaderHash 128720366063791916))))
          TraceAddBlockEvent (AddBlockValidation (ValidCandidate (AnchoredSeq {anchor = Anchor (SlotNo 0) (TestHeaderHash 2173152789646624309) (BlockNo 0), unanchorSeq = SFT {fromStrict = fromList [MeasuredWith {unMeasuredWith = TestHeader {thHash = TestHeaderHash (-2651776708921877814), thPrevHash = BlockHash (TestHeaderHash 2173152789646624309), thBodyHash = TestBodyHash 590682968308805179, thSlotNo = SlotNo 0, thBlockNo = BlockNo 1, thChainLength = ChainLength 2, thIsEBB = RegularBlock}}]}})))

          PostconditionFailed "AnnotateC \"real response didn't match model response\" (PredicateC (Resp {getResp = Right (IsValid (IsValidResult {real = True, isValid = Just True}))} :/= Resp {getResp = Right (IsValid (IsValidResult {real = False, isValid = Nothing}))}))" /= Ok
          Use --quickcheck-replay=455411 to reproduce.
          Use -p '/ChainDB q-s-m.sequential/' to rerun this test only.

1 out of 40 tests failed (103.73s)

amesgen avatar Apr 04 '22 15:04 amesgen

With #3743, this will no longer annoy us in CI for the time being, but this should be revisited in the future.

amesgen avatar May 12 '22 15:05 amesgen

Difference in behaviour

The difference in behaviour between the actual API implementation and the model with respect to the valid and invalid states in described by tables below, which depict the states in the chain selection algorithms, the first for the actual implementation, the second for the model, (note that invalid states are indicated by lowercase letters, while valid states are indicated by uppercase letters).

Block Candidates Seen (diff) Selected
C :> d [GABCd] d
[GABC] GABC
A :> E [GABC, GAE] GABC
E :> f [GABC, GAEf]
[GAEf, GABC] E, f
[GABC, GAE] GABC
Block Candidates Seen (diff) Selected
C :> d [GABCd] d
[GABC] GABC
A :> E [GABCd, GAE]
[GABC, GAE] GABC
E :> f [GABCd, GAEf]
[GABC, GAE] GABC

There is never a stage in which the GAEf chain is a plausible candidate in the model, due to the fact that it does simultaneous pruning of invalid blocks, before it does selection. It might be difficult to adapt this implementation of the model to precisely match that of the actual one, with respect to the valid and invalid states.

getIsValid not used anywhere

Another point is that the getIsValid function in the ChainDB API is not used anywhere, not in ouroboros-consensus, nor cardano-node, so it might be better to remove it completely, as it also seems to expose something that is really an optimization in the internals of the ChainDB. In that case, we could simply remove GetIsValid from Cmd in Test.Ouroboros.Storage.ChainDB.

bartfrenk avatar Aug 17 '22 12:08 bartfrenk

The team discussed this today, and came to the following conclusions:

  1. Reduce the ChainDB API by making getIsValid redundant: #3974
  2. We agreed that not reverting #3743 in some way leaves room for degenerate behavior to sneak in: For example, if due to a bug, the model implementation never returns Just, every behavior of the actual getIsValid is accepted.
  3. There are multiple paths forward for this ticket:
    1. Modify the model implementation in such a way that we can revert #3743 (probably make it more "incremental"), while keeping it sufficiently simple/different from the actual ChainSel.
    2. Continue to only check agreement when both return Just, but add some additional sanity checks that getIsValid is not completely misbehaving as described in 2. (i.e. it should always know that the blocks in our selection are valid).
    3. Find a different way to test the behavior of ChainDB tracking invalid blocks that we rely on in the ChainSync client and ChainSel.

amesgen avatar Aug 23 '22 14:08 amesgen

I took a closer look today.

I think this takeWhile might be the problem. I anticipate that Javier wrote that because validChains is careful to sort the chains before it processes them via foldMap, which preserves that ordering. However, I wonder if that order, since it is from before the detection of invalid chains, it correct here.

If I change this takeWhile to filter, then the repro passes. I'm ~not~ now running the test suite several times; I'll report back in a while.

https://github.com/input-output-hk/ouroboros-network/blob/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa/ouroboros-consensus-test/test-storage/Test/Ouroboros/Storage/ChainDB/Model.hs#L450-L453

Edit: after running the test suite ~100 times, I did see an error. Looks unfamiliar.

PostconditionFailed
 "AnnotateC \"real response didn't match model response\" (
 
 PredicateC
   (Resp {getResp = Right (IterResultGCed (IteratorResultGCed {real = True, iterResult = IteratorExhausted}))}
      :/= Resp {getResp = Right (IterResultGCed (IteratorResultGCed {real = False, iterResult = IteratorResult (TestBlock {testHeader = TestHeader {thHash = TestHeaderHash (-340232222464533890), thPrevHash = BlockHash (TestHeaderHash 8656999388501844387), thBodyHash = TestBodyHash 590680769285548757, thSlotNo = SlotNo 4, thBlockNo = BlockNo 2, thChainLength = ChainLength 3, thIsEBB = RegularBlock}, testBody = TestBody {tbForkNo = 3, tbIsValid = True}},TestBlock {testHeader = TestHeader {thHash = TestHeaderHash (-340232222464533890), thPrevHash = BlockHash (TestHeaderHash 8656999388501844387), thBodyHash = TestBodyHash 590680769285548757, thSlotNo = SlotNo 4, thBlockNo = BlockNo 2, thChainLength = ChainLength 3, thIsEBB = RegularBlock}, testBody = TestBody {tbForkNo = 3, tbIsValid = True}},TestHeader {thHash = TestHeaderHash (-340232222464533890), thPrevHash = BlockHash (TestHeaderHash 8656999388501844387), thBodyHash = TestBodyHash 590680769285548757, thSlotNo = SlotNo 4, thBlockNo = BlockNo 2, thChainLength = ChainLength 3, thIsEBB = RegularBlock},\"\\131\\NUL\\136\\NUL;\\EOT\\184\\191gY#\\229\\129\\129\\ESCx#\\215\\DC3\\188m!\\163\\ESC\\b2\\133\\a\\180\\235j\\213\\EOT\\STX\\ETX\\129\\SOH\\131\\NUL\\ETX\\245\",\"\\136\\NUL;\\EOT\\184\\191gY#\\229\\129\\129\\ESCx#\\215\\DC3\\188m!\\163\\ESC\\b2\\133\\a\\180\\235j\\213\\EOT\\STX\\ETX\\129\\SOH\",TestHeaderHash (-340232222464533890),SlotNo 4,IsNotEBB,41,35,SomeSecond (NestedCtxt {flipNestedCtxt = CtxtTestBlock}))}))}))"
 /= Ok

Edit: After 325 executions of the test suite, the above is still the only error I saw, with filter instead of takeWhile.

nfrisby avatar Sep 08 '22 02:09 nfrisby

^^^ I opened Issue #3999 for the above failure

cc @bartfrenk

nfrisby avatar Sep 08 '22 21:09 nfrisby