PoS testing: InvalidChainFound: invalid block=fc2b3720bcd3584a1ded5090a9babed683173d98590cb4c4143bff89e56bb4fd height=33123
Summary
A node (which may have seized up and been restarted) ended up on the wrong chain, banned by it's peers and unable to sync.
Steps to reproduce
Unknown.
Expected behavior
Nodes should not generate/propagate invalid blocks.
Problematic behavior
A node found itself on an invalid chain and couldn't get off. The invalid chain should not have been created in the first place.
Crown-core environment info:
Ubuntu 16.04
Crown-core application info
Crown version v0.12.7.3-2f1de615
Relevant logs, dumps and/or screenshots
2019-02-05 07:39:23 ProcessNewBlock : ACCEPTED
2019-02-05 07:39:23 ProcessNewBlock : ACCEPTED
2019-02-05 07:39:25 CActiveMasternode::SendMasternodePing() - Relay Masternode Ping vin = CTxIn(COutPoint(5b33dbec20784166f310f6059e719fe4d2622c71cbee183b4fe739eb034192b4, 1), scriptSig=)
2019-02-05 07:40:50 Misbehaving: 23.95.16.96:19340 (0 -> 1)
2019-02-05 07:40:50 ProcessMessage(reject, 42 bytes) FAILED peer=1146
2019-02-05 07:40:50 Misbehaving: 23.95.16.96:19340 (1 -> 2)
2019-02-05 07:40:50 ProcessMessage(ping, 8 bytes) FAILED peer=1146
2019-02-05 07:41:32 peer=1147 using obsolete version 71054; disconnecting
2019-02-05 07:41:32 ProcessMessage(version, 107 bytes) FAILED peer=1147
2019-02-05 07:41:36 peer=1148 using obsolete version 71054; disconnecting
2019-02-05 07:41:36 ProcessMessage(version, 107 bytes) FAILED peer=1148
2019-02-05 07:43:43 ERROR: CheckBlockProofPointer: Stake pointers cannot be from budget blocks
2019-02-05 07:43:43 ERROR: CheckStake: Invalid block proof pointer
2019-02-05 07:43:43 ERROR: ConnectBlock: Block has invalid proof of stake
2019-02-05 07:43:43 Misbehaving: 80.211.161.253:19340 (0 -> 100) BAN THRESHOLD EXCEEDED
2019-02-05 07:43:43 InvalidChainFound: invalid block=fc2b3720bcd3584a1ded5090a9babed683173d98590cb4c4143bff89e56bb4fd height=33123 log2_work=42.81867 date=2019-02-05 07:44:14
2019-02-05 07:43:43 InvalidChainFound: current best=72a0d96ffcbf8bd9a6cbe85f4d9741cf6d69c219582bbe2592ab26db804c3432 height=33122 log2_work=42.818581 date=2019-02-05 07:39:54
2019-02-05 07:43:43 ERROR: ConnectTip() : ConnectBlock fc2b3720bcd3584a1ded5090a9babed683173d98590cb4c4143bff89e56bb4fd failed
e019-02-05 07:43:43 InvalidChainFound: invalid block=fc2b3720bcd3584a1ded5090a9babed683173d98590cb4c4143bff89e56bb4fd height=33123 log2_work=42.81867 date=2019-02-05 07:44:14
2019-02-05 07:43:43 InvalidChainFound: current best=72a0d96ffcbf8bd9a6cbe85f4d9741cf6d69c219582bbe2592ab26db804c3432 height=33122 log2_work=42.818581 date=2019-02-05 07:39:54
2019-02-05 07:43:43 ProcessNewBlock : ACCEPTED
2019-02-05 07:43:47 peer=1150 using obsolete version 71054; disconnecting
2019-02-05 07:43:47 ProcessMessage(version, 107 bytes) FAILED peer=1150
2019-02-05 07:44:12 peer=1151 using obsolete version 70053; disconnecting
2019-02-05 07:44:12 ProcessMessage(version, 116 bytes) FAILED peer=1151
2019-02-05 07:44:18 peer=1152 using obsolete version 70053; disconnecting
2019-02-05 07:44:18 ProcessMessage(version, 116 bytes) FAILED peer=1152
2019-02-05 07:44:25 CActiveMasternode::SendMasternodePing() - Relay Masternode Ping vin = CTxIn(COutPoint(5b33dbec20784166f310f6059e719fe4d2622c71cbee183b4fe739eb034192b4, 1), scriptSig=)
2019-02-05 07:44:52 ERROR: AcceptBlockHeader : prev block (fc2b3720bcd3584a1ded5090a9babed683173d98590cb4c4143bff89e56bb4fd) invalid
2019-02-05 07:44:52 ERROR: ProcessNewBlock : AcceptBlock FAILED
2019-02-05 07:44:52 Misbehaving: 37.139.28.90:19340 (0 -> 100) BAN THRESHOLD EXCEEDED
2019-02-05 07:44:52 ERROR: AcceptBlockHeader : prev block (fc2b3720bcd3584a1ded5090a9babed683173d98590cb4c4143bff89e56bb4fd) invalid
2019-02-05 07:44:52 ERROR: ProcessNewBlock : AcceptBlock FAILED
2019-02-05 07:44:52 Misbehaving: 164.132.151.94:19340 (0 -> 100) BAN THRESHOLD EXCEEDED
2019-02-05 07:44:52 ERROR: AcceptBlockHeader : prev block (fc2b3720bcd3584a1ded5090a9babed683173d98590cb4c4143bff89e56bb4fd) invalid
2019-02-05 07:44:52 ERROR: ProcessNewBlock : AcceptBlock FAILED
2019-02-05 07:44:52 Misbehaving: 81.92.203.215:39138 (0 -> 100) BAN THRESHOLD EXCEEDED
2019-02-05 07:44:52 ERROR: AcceptBlockHeader : prev block (fc2b3720bcd3584a1ded5090a9babed683173d98590cb4c4143bff89e56bb4fd) invalid
...Uploaded 20190205_walkjivefly_MN_PMN01_invalid_chain.log to https://nextcloud.crown.tech/nextcloud/s/znd8HiiAsRX3C6B
Possible fixes
(Any comments on what you think might be responsible for the problem, if you have particular insight.)
/cc @artem
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information