cncli is out of sync with cardano node?

[monkey-jsun]

cncli leader log reported a block for today. However my node did not produce it.

Later on, I deleted cncli db and re-sync db from scratch. After that I ran leaderlog again. I got different nonce values and got no block prediction. Apparently this nonce discrepancy happened since last epoch. See more notes below.

What could be causing this discrepancy between cncli and cardano node? How can we detect and prevent this? Today's case is good one in the sense it is false positive. If it is a false negative and we miss the block, it will be kind of expensive.

thanks.

====================
before re-sync:
{
  "status": "ok",
  "epoch": 302,
  "epochNonce": "546d96071e0ec473b9d0d6ac73ad06595e9f3daf34fc7e3915c9c19789ac2740",
  "epochSlots": 1,
  "epochSlotsIdeal": 0.16,
  "maxPerformance": 625.0,
  "poolId": "66bd46cec6924459230b7bd70b109c6424804a8f6abd8d3855ed043d",
  "sigma": 7.300612356436153e-6,
  "activeStake": 174109183728,
  "totalActiveStake": 23848572589189307,
  "d": 0.0,
  "f": 0.05,
  "assignedSlots": [
    {
      "no": 1,
      "slot": 45253912,
      "slotInEpoch": 153112,
      "at": "2021-11-13T08:16:43-08:00"
    }
  ]
}

epoch 301
{
  "status": "ok",
  "epoch": 301,
  "epochNonce": "0bbf3e14bf4a232142435c6c31e07df50e284b3b9cf314b2d9a45506f19f9779",
  "epochSlots": 0,
  "epochSlotsIdeal": 0.16,
  "maxPerformance": 0.0,
  "poolId": "66bd46cec6924459230b7bd70b109c6424804a8f6abd8d3855ed043d",
  "sigma": 7.297631047990907e-6,
  "activeStake": 174103268044,
  "totalActiveStake": 23857504839454979,
  "d": 0.0,
  "f": 0.05,
  "assignedSlots": []
}

=====================
after a complete re-sync:

{
  "status": "ok",
  "epoch": 302, 
  "epochNonce": "cb3aef21a34dbb2807aa508e03f1d35cffdcf98114d22a193bf8148983522711",
  "epochSlots": 0,
  "epochSlotsIdeal": 0.16,
  "maxPerformance": 0.0,
  "poolId": "66bd46cec6924459230b7bd70b109c6424804a8f6abd8d3855ed043d",
  "sigma": 7.300612356436153e-6,
  "activeStake": 174109183728,
  "totalActiveStake": 23848572589189307,
  "d": 0.0,
  "f": 0.05,
  "assignedSlots": []
}

{
  "status": "ok",
  "epoch": 301, 
  "epochNonce": "c0e8aa015de7703c6fbec6c85a0aafb0974082e1eb4808061ad2e5ef23a2fd62",
  "epochSlots": 0,
  "epochSlotsIdeal": 0.16,
  "maxPerformance": 0.0,
  "poolId": "66bd46cec6924459230b7bd70b109c6424804a8f6abd8d3855ed043d",
  "sigma": 7.297631047990907e-6,
  "activeStake": 174103268044,
  "totalActiveStake": 23857504839454979,
  "d": 0.0,
  "f": 0.05,
  "assignedSlots": []
}

[monkey-jsun]

I should add that I'm using systemd service to maintain the sync. See the script below.

[Unit]
Description=CNCLI Sync
After=multi-user.target

[Service]
Type=simple
Restart=always
RestartSec=5
User=ubuntu
Group=ubuntu
LimitNOFILE=131072
ExecStart=/home/ubuntu/cncli/cncli sync --host localhost --db /home/ubuntu/cncli/cardano-cncli.db
KillSignal=SIGINT
SuccessExitStatus=143
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=cncli-sync

[Install]
WantedBy=multi-user.target

[AndrewWestberg]

I've seen this happen when people sync the same db against two networks like testnet and mainnet. I've never seen it in my own instances running sync. The next time it happens, can you attach the bad database file to the ticket so I can check it?

[monkey-jsun]

Sure. Will do.

Meanwhile is there a way for me to periodically check, detect and catch any out-of-sync happening? This time it was purely accidental.