What happened?

August 22 06:48:

Superblock #3 was not paid

October 24 04:02:

Superblock #5 was not paid

November 25 14:07:

Superblock #6 was paid, but the team decided to vote manually on finalized budget

December 26 02:38:

Superblock #7 ended up not having been paid due to the fork in the blockchain. In fact, on the original chain the superblock was paid, although manual voting did occur

(UPD) February 28 12:02:

Superblock #9 was paid, but needed manual voting for finalized budget

Why it happened?

Superblock #3

Technically, the superblock failed to be paid because no finalized budget was submitted during budget finalization period. Apparently, the node that was to submit the finalized budget had crashed, but there is a human component to what happened as well. The only person who knew at the moment anything about how governance works didn't pay attention to the signs of trouble, and the rest of the team didn't had the knowledge.

Superblock #5

This time there was at least two finalized budget versions in the network, but none of them got enough votes to pass the threshold (probably, both got zero votes). This happened because none of the finalized budgets that were submitted by the node responsible for that (Fabian's wallet) coincided with the budget that masternodes had. The reasons of why it can happen are described below.

Superblock #6

There were at least 5 different versions of finalized budget on the network. November 23 starting after 15:45 (approximately 23 hours before the supreblock) one of them, ea79101e..., started to receive manual votes ending up with 274 votes. It's not possible to say how many votes were given automatically or how many votes it would have received without manual intervention. The runner-up, an older one, 73fdbde6..., received automatic 45 votes which was at the brink of passing.

Though it is not certain, it is possible that the budget for Superblock #6 might have been finalized automatically

Superbock #7

There were at least 3 different version of finalized budget with 188a096f... having 99 votes before it was voted manually for (ending up with 155 votes). Even without manual voting the budget would have been paid.

Sadly, because of the fork that was happening at that time none of it mattered.

(UPD) Superblock #9

Not unlike the October superblock, there were several finalized budgets on the network (9 budgets, to be precise), but neither of them received the amount of votes required. In fact all of them received 0 votes and just one budget received 4. In the end the team voted manually for one of the budget with their masternodes

Conclusion

From the 8 superblocks that happened during the lifetime of Crown there were 3 incidents were the budget payment system failed to function. Reasons for superblock #3 and superblock #7 not having been paid are clear.

Superblock #5 is the most interesting case, which we cannot fully explain despite the long investigation. Similar situation was observed during superblock #9. What we do know about why it failed (as well as the suggestions for further actions) is described below.

How Governance work?

1. Proposal submission is a two-step process:

   1. First, you have to "prepare" the proposal with mnbudget prepare. It creates a transaction burning 25 CRW and containing the hash of the not-yet-created proposal
   2. After the transaction has 6 confirmations, you can submit the proposal with mnbudget submit. It will be broadcasted into the network. It doesn't have to be for the next superblock, it can be for any of the superblocks in the future

2. A masternode owner can vote for proposals from the wallet that controls their masternodes, a vote per masternode. For each proposal votes on behalf of the same masternode can be issued with at least 1 hour interval.

3. Each node on the network keeps a container of all proposals (including old proposals or proposals for future superblocks). The entire list can be viewed with mnbudget show. From this set of proposals a subset is selected, following this rules:

   • All proposals are sorted by difference between "yes" and "no" votes
   • Only proposals for the next superblock are selected
   • Amount of "yes" minus "no" votes must be greater than 10% amount of masternodes currently online
   • Proposal must be at least 24 hours old, i.e. "established" (therefore, a proposal that is created less a day before the superblock has zero chance of being included)
   • From the sequence ordered and filtered as described above top X proposals are picked where X is such that Σ amount(X) <= 43200 CRW

   This subset is considered to be the node's view of the budget for the next superblock. It can be viewed with mnbudget projection. This budget continuously evolves as new proposals and votes are being broadcast through the network

4. 2880 blocks before the superblock starts budget finalization. This process consists of one node submitting finalized budget and all the masternodes voting for that budget. The node that will submit finalized budgets must to be run with -budgetvotemode=suggest command line argument (or with the corresponding line in the config file). It is again a two-step process

   1. Every time it receives a new block, will check if its current view of the budget was submitted as finalized budget. If no, the node will create a transaction burning 25 CRW and containing the hash of the finalized budget to be created
   2. After 7 (!) confirmations of the collateral transaction the node broadcasts finalized budget into the network. If before 7th confirmation arrived the view of the budget is changed the process starts again (this is why we typically see 25 CRW being paid more times than the number of submitted budgets)

   The node responsible for budget finalization (it's been Fabian's wallet since September) needs to be online during finalization process (2880 blocks) and to have enough funds in order to be able to submit finalized budgets. There might be several nodes submitting finalized budgets.

   The finalized budgets can be viewed with mnfinalbudget show command on any node.

5. When a masternode receives finalized budget broadcast it compares it with it's own view of the budget. If both are identical (i.e. have the same proposals in the same order), it will vote for it and broadcast the vote into the network. If somehow a masternode receives finalized budget before all the proposal votes, it won't issue votes, will mark the finalized budget as checked and won't return to it ever (this will be changed for 12.4)

6. During the budget finalization it is still possible to vote for the proposals. When votes are broadcast, they reach the node that submits finalized budget at some point. If it triggers a change in the budget (either proposal set or the ordering in that set), a new finalized budget is broadcast and voted for. Typically, it looks like this: there is a new finalized budget, it receives votes, it stops receiving votes, a there is a new finalized budget, repeat.

7. When the mining node starts creating the superblock it will get the finalized budget with the highest amount of votes that it knows of and will put the payment for the corresponding proposal into the block. The proposals are paid out one per block: if the superblock height is N than the first proposal is paid in block N, the next one is paid in block N+1, and so on. The miner will only pay out a finalized budget that has more votes than 0.05 * M, where M is the number of masternodes currently online.

   If the mining node goes online before it finishes masternode synchronization process, it won't check the finalized budgets against the 5% M threshold, it will just pay the budget with highest number of votes.

8. All nodes receiving that block validate it and, as a part of this process they check the budget payment transactions:

   • The node considers a budget payment transaction invalid if it has less than 0.05 * M votes, where M is the number of masternodes currently online.
   • The node accepts transactions for all finalized budgets that have X votes, where X is such that it belongs to confidence interval F-0.1M < X < F+0.1M, where F equals the highest amount of votes for all finalized budgets for this superblock.

Where it can go wrong?

There are several places where budget system can fail:

1. No finalized budget is submitted (this is what happened with superblock #3)
2. Finalized budget is submitted, but didn't reach enough masternodes (we saw no evidence of that having place)
3. There is one or several finalized budgets and a good portion of masternodes received them, but the budget did not get enough votes (this is what happened with superblock #5 and there was a chance of this happening with superblock #6). There can be several reasons for that:
   • Some of the masternodes didn't receive all votes for proposals and therefore have different view of the currnet budget than. In fact, we repeatedly saw that it took long time for votes to propagate through network
   • Some of the masternodes received finalized budget earlier than some of the votes that the node submitting that final budget had
   • Compliant nodes were accused of misbehavior and banned due to a software error which made some portion of votes difficult or impossible to reach all masternodes
   • Network failure or inefficiencies in the node graph may cause votes to propagate slowly
4. Masternodes are configured with budgetvotemode being equal to anything else then auto (we have no reasons to believe this to ever have been the case)
5. Enough masternodes voted for one of the finalized budgets, but the votes (or the budget to that matter) did not reach the mining node that gets to issue the superblock (among other things, this happened during superblock #7)
6. Although it's unlikely, it's possible that a mining node issuing blocks with budget payments will receive finalized budget votes before it finishes the payments. In this case it might start making payments for one budget and finish making payments for another. The same thing is possible if two miners have different vote count for finalized budgets during payments (this never happened, apparently)

What should we do next?

The governance system that we inherited from Dash has some design and implementation flaws that need to be addressed during the next few months. Here are improvements that I suggest:

• The need to manually run the node that submits finalized budgets is a constant source of possibilities for human errors (although since the moment Fabian is in charge of running that node it has never been the case). Dash solved this with their sentinel subsystem, we also need to come up with something. We don't have a specific solution at the moment, but we will.
• Budget finalization is somewhat expensive. Each attempt costs 25 CRW to Fabian and sometimes there were more than a dozen attempts per each budget (13 was the highest confirmed number of attempts)
• The fact that comparing or paying out budgets depends on the relative order of the proposals that made it into the budget increases the chances for disagreement and the amount of finalized budgets broadcast into the network (and consequently, the amount of money taken from Fabian's wallet). I suggest that proposals that made it into the budget are to be compared regardless of their order and paid in order that doesn't depend on the actual votes (or, alternatively, paid in a single block).
• We can forbid voting for proposals 2880 blocks before the superblock. I suspect it to also reduce the amount of disagreement during budget finalization dramatically. It can be done independently at two levels:
  • at interface level when creating new votes (this can be done very quickly)
  • at block validation level (this is a more deep change as it changes the rules blockchain is validated against)
• We will need to find out the cause of proposal votes being propagated through network too slowly at some point anyway
• May be questionable, but depending on how deep we intend to change Bitcoin protocol we might think of including all information about the proposals and votes into the blockchain. This way the consensus on the chain will automatically mean the consensus on the budget (or any governance decision for that matter). There will be no need for dedicated budget finalization by masternodes.