Timeline

Time	Events
19.01 08:35	Proposal test8 for 42 tCRW starting block 57750 is submitted by Volodymyr
19.01 08:35	Block 57664 is mined. It includes 25 tCRW fee for test8 submission
19.01 08:44	Proposal test8 receives 2 votes
19.01 09:39	Final budget A is submitted by one of vshamray's nodes. The only proposal it includes is test8
19.01 09:39	Block 57721 is mined. It includes 25 tCRW as final budget submission fee
19.01 09:55	Final budget A receives 3 votes from masternodes
19.01 10:25	Block 57749 is mined. Mining stops until late in the evening
19.01 19:23	Block 57750(A) is mined. It contains 42 tCRW payment by proposal test8 (final budget A)
19.01 19:24	Block 57750(A) reaches Volodymyr's masternodes. Both nodes reject it as it doesn't contain payment to DevFund Address
19.01 19:27	Masternodes keep rejecting the block, so does (apparently) testnet explorer
20.01 10:24	Block 57750(B) is mined (how? by which node?). It contains 50 tCRW payment to DevFund (failsafe) and does not contain payment by test8 proposal
22.01	Both Volodymyr's masternodes have full information about test8 and its votes. The consensus on budget seems to be at final budget A, yet the blockchain is on chain B

Hash reference

Object	Hash
block 57664	00000000055297a028ed7f74f05de37e49d6f5e3b56a1f5744a1daaec7828737
block 57721	000000001453978d77294ebadf1cbb1d4fc7174059e9407072a671e7e9684bdc
block 57749	00000000112defd05ff138031ff5bff7a4967118c6acdb1d03541d356ef58804
block 57750(A)	00000000ff4cc3d7506c5b48ab3aca7aea2da0e1876494187dd99b6f9090f8ef
block 57750(B)	000000002856e18e100e7c56d5486fcbd537a4a87da2e5b393409d046393c9c0
test8	560297e695d6615ca5f1c861d1c91b0b04be9c76fa9404300fa6689037eb692b
Final Budget A	1f2995bdd6597a4fc5ba9f5a4813d14174733c512369c4adf0d2821a215a8526

Address reference

Label	Address
DevFund Address	mr59c3aniaN3qHXej5L8UBsssRZbiUUMnz

Interpretation

The fork happened because of a bug in validating superblock payments that only reproduces with small amount (< 20) of masternodes in the network (which is exactly the case for the testnet). Hence, block 57750(A) that contained budget payment was rejected by several nodes in the network; later those nodes accepted block 57750(B) that contained failsafe payment.

Due to sloppy use of integer arithmetics in function CBudgetManager::IsTransactionValid superblock payment validity checks of all nodes that were online and synced at the moment (except for Volodymyr's wallet which was patched) rejected block 57750(A) by error. Mining pool node, however, produced the block before it had time to sync, so it still paid the budget that it had saved, but didn't run any masternode or budget related checks. Later, all the nodes that received block 57750(A) before they finished syncing ended up on chain A, others ended up on chain B.

Therefore, following 3 events combined together resulted in the fork:

1. Calculation bug in CBudgetManager::IsTransactionValid
2. Small amount of masternodes in the testnet (which is the condition for the bug to reproduce)
3. Mining pool node having gone online exactly before the superblock (before it had time to sync the masternode list) - otherwise it wouldn't have created the block because of the bug mentioned above

Actions taken

Following steps aim to test the hypothesis and fix the problem:

1. An update that fixes calculations in IsTransactionValid was produced.
2. The update was deployed into testnet
3. The miner continued to mine on chain A and all updated nodes in the network switched to that chain