Block corruption still occurring with pipeline #848 build
Summary
crown@2093:~$ crown-cli stop
Crown server stopping
crown@2093:~$ crownd
Crown server starting
crown@2093:~$ : Corrupted block database detected.
Do you want to rebuild the block database now?Steps to reproduce
As above, but doesn't happen every time.
Expected behavior
Shutdown, restart should not corrupt the block database.
Problematic behavior
Block database corrupted on shutdown
Crown-core environment info
MX 17.1 linux
Crown-core application info
Crown version v0.12.5.1-5184c558 (2018-12-07 13:59:07 -0700)
Relevant logs and/or screenshots
2018-12-12 03:05:40 StopNode()
2018-12-12 03:05:40 Verifying mncache.dat format...
2018-12-12 03:05:40 Reading info from mncache.dat...
2018-12-12 03:05:40 ERROR: ReadStream: Failed to open file /home/crown/.crown/testnet3/mncache.dat
2018-12-12 03:05:40 Missing file mncache.dat, will try to recreate
2018-12-12 03:05:40 Writing info to mncache.dat...
2018-12-12 03:05:40 Written info to mncache.dat 0ms
2018-12-12 03:05:40 Masternodes: 1, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-12-12 03:05:40 mncache.dat dump finished 1ms
2018-12-12 03:05:40 Verifying budget-v2.dat format...
2018-12-12 03:05:40 Reading info from budget-v2.dat...
2018-12-12 03:05:40 ERROR: ReadStream: Failed to open file /home/crown/.crown/testnet3/budget-v2.dat
2018-12-12 03:05:40 Missing file budget-v2.dat, will try to recreate
2018-12-12 03:05:40 Writing info to budget-v2.dat...
2018-12-12 03:05:40 Written info to budget-v2.dat 0ms
2018-12-12 03:05:40 Proposals: 0, Budgets: 0, Seen Budgets: 0, Seen Budget Votes: 0, Seen Final Budgets: 0, Seen Final Budget Votes: 0
2018-12-12 03:05:40 budget-v2.dat dump finished 0ms
2018-12-12 03:05:40 Verifying mnpayments.dat format...
2018-12-12 03:05:40 Reading info from mnpayments.dat...
2018-12-12 03:05:40 ERROR: ReadStream: Failed to open file /home/crown/.crown/testnet3/mnpayments.dat
2018-12-12 03:05:40 Missing file mnpayments.dat, will try to recreate
2018-12-12 03:05:40 Writing info to mnpayments.dat...
2018-12-12 03:05:40 Written info to mnpayments.dat 5ms
2018-12-12 03:05:40 Votes: 716, Blocks: 179
2018-12-12 03:05:40 mnpayments.dat dump finished 6ms
2018-12-12 03:05:40 Verifying sncache.dat format...
2018-12-12 03:05:40 Reading info from sncache.dat...
2018-12-12 03:05:40 ERROR: ReadStream: Failed to open file /home/crown/.crown/testnet3/sncache.dat
2018-12-12 03:05:40 Missing file sncache.dat, will try to recreate
2018-12-12 03:05:40 Writing info to sncache.dat...
2018-12-12 03:05:40 Written info to sncache.dat 0ms
2018-12-12 03:05:40 Systemnodes: 0, peers who asked us for Systemnode list: 0, peers we asked for Systemnode list: 0, entries in Systemnode list we asked for: 0
2018-12-12 03:05:40 sncache.dat dump finished 2ms
2018-12-12 03:05:40 Verifying snpayments.dat format...
2018-12-12 03:05:40 Reading info from snpayments.dat...
2018-12-12 03:05:40 ERROR: ReadStream: Failed to open file /home/crown/.crown/testnet3/snpayments.dat
2018-12-12 03:05:40 Missing file snpayments.dat, will try to recreate
2018-12-12 03:05:40 Writing info to snpayments.dat...
2018-12-12 03:05:40 Written info to snpayments.dat 2ms
2018-12-12 03:05:40 Votes: 73, Blocks: 49
2018-12-12 03:05:40 snpayments.dat dump finished 4ms
2018-12-12 03:05:40 Verifying ixcache.dat format...
2018-12-12 03:05:40 Reading info from ixcache.dat...
2018-12-12 03:05:40 ERROR: ReadStream: Failed to open file /home/crown/.crown/testnet3/ixcache.dat
2018-12-12 03:05:40 Missing file ixcache.dat, will try to recreate
2018-12-12 03:05:40 Writing info to ixcache.dat...
2018-12-12 03:05:40 Written info to ixcache.dat 10ms
2018-12-12 03:05:40 Transaction lock requests: 0, Transaction locks: 0, Locked Inputs: 0, Transaction lock votes: 495
2018-12-12 03:05:40 ixcache.dat dump finished 12ms
2018-12-12 03:05:40 Shutdown: done
2018-12-12 03:05:48
2018-12-12 03:05:48 Crown version v0.12.5.1-5184c55 (2018-12-07 13:59:07 -0700)
2018-12-12 03:05:48 Using OpenSSL version OpenSSL 1.0.1k 8 Jan 2015
2018-12-12 03:05:48 Using BerkeleyDB version Berkeley DB 4.8.30: (April 9, 2010)
2018-12-12 03:05:48 Default data directory /home/crown/.crown
2018-12-12 03:05:48 Using data directory /home/crown/.crown/testnet3
2018-12-12 03:05:48 Using config file /home/crown/.crown/crown.conf
2018-12-12 03:05:48 Using at most 125 connections (1024 file descriptors available)
2018-12-12 03:05:48 Using 0 threads for script verification
2018-12-12 03:05:48 Binding RPC on address :: port 19341 (IPv4+IPv6 bind any: 1)
2018-12-12 03:05:48 Creating backup of "/home/crown/.crown/testnet3/wallet.dat" -> "/home/crown/.crown/testnet3/backups/wallet.dat.2018-12-12-03-05"
2018-12-12 03:05:48 Using wallet wallet.dat
2018-12-12 03:05:48 init message: Verifying wallet...
2018-12-12 03:05:48 CDBEnv::Open: LogDir=/home/crown/.crown/testnet3/database ErrorFile=/home/crown/.crown/testnet3/db.log
2018-12-12 03:05:48 Bound to 196.52.70.4:19340
2018-12-12 03:05:48 AddLocal(196.52.70.4:19340,4)
2018-12-12 03:05:48 Cache configuration:
2018-12-12 03:05:48 * Using 37.5MiB for block index database
2018-12-12 03:05:48 * Using 8.0MiB for chain state database
2018-12-12 03:05:48 * Using 254.5MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space)
2018-12-12 03:05:48 init message: Loading block index...
2018-12-12 03:05:48 Opening LevelDB in /home/crown/.crown/testnet3/blocks/index
2018-12-12 03:05:49 Opened LevelDB successfully
2018-12-12 03:05:49 Opening LevelDB in /home/crown/.crown/testnet3/chainstate
2018-12-12 03:05:49 Opened LevelDB successfully
2018-12-12 03:05:49 LoadBlockIndexDB: last block file = 0
2018-12-12 03:05:49 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=13771, size=4802416, heights=0...13649, time=2018-09-14...2018-12-11)
2018-12-12 03:05:49 Checking all blk files are present...
2018-12-12 03:05:49 LoadBlockIndexDB(): transaction index enabled
2018-12-12 03:05:49 LoadBlockIndexDB(): hashBestChain=99bf058cb54c2e2c04c072fc448d35d21198366169cce2b4ba323fead12399f9 height=13649 date=2018-12-11 10:12:50 progress=0.868425
2018-12-12 03:05:49 init message: Verifying blocks...
2018-12-12 03:05:49 Verifying last 288 blocks at level 3
2018-12-12 03:05:49 ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
2018-12-12 03:05:49 ERROR: VerifyDB() : *** coin database inconsistencies found (last 40 blocks, 620 good transactions before that)
2018-12-12 03:05:49 : Corrupted block database detected.
Do you want to rebuild the block database now?
2018-12-12 03:05:49 Aborted block database rebuild. Exiting.
2018-12-12 03:05:49 PrepareShutdown: In progress...
2018-12-12 03:05:49 RPCAcceptHandler: Error: Operation canceled
2018-12-12 03:05:49 StopNode()
2018-12-12 03:05:49 Verifying mncache.dat format...
2018-12-12 03:05:49 Reading info from mncache.dat...
2018-12-12 03:05:49 Loaded info from mncache.dat 0ms
2018-12-12 03:05:49 Masternodes: 1, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-12-12 03:05:49 Writing info to mncache.dat...
2018-12-12 03:05:49 Written info to mncache.dat 0ms
2018-12-12 03:05:49 Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-12-12 03:05:49 mncache.dat dump finished 1ms
2018-12-12 03:05:49 Verifying budget-v2.dat format...
2018-12-12 03:05:49 Reading info from budget-v2.dat...
2018-12-12 03:05:49 Loaded info from budget-v2.dat 0ms
2018-12-12 03:05:49 Proposals: 0, Budgets: 0, Seen Budgets: 0, Seen Budget Votes: 0, Seen Final Budgets: 0, Seen Final Budget Votes: 0
2018-12-12 03:05:49 Writing info to budget-v2.dat...
2018-12-12 03:05:49 Written info to budget-v2.dat 1ms
2018-12-12 03:05:49 Proposals: 0, Budgets: 0, Seen Budgets: 0, Seen Budget Votes: 0, Seen Final Budgets: 0, Seen Final Budget Votes: 0
2018-12-12 03:05:49 budget-v2.dat dump finished 1ms
2018-12-12 03:05:49 Verifying mnpayments.dat format...
2018-12-12 03:05:49 Reading info from mnpayments.dat...
2018-12-12 03:05:49 Loaded info from mnpayments.dat 0ms
2018-12-12 03:05:49 Votes: 716, Blocks: 179
2018-12-12 03:05:49 Writing info to mnpayments.dat...
2018-12-12 03:05:49 Written info to mnpayments.dat 0ms
2018-12-12 03:05:49 Votes: 0, Blocks: 0
2018-12-12 03:05:49 mnpayments.dat dump finished 4ms
2018-12-12 03:05:49 Verifying sncache.dat format...
2018-12-12 03:05:49 Reading info from sncache.dat...
2018-12-12 03:05:49 Loaded info from sncache.dat 0ms
2018-12-12 03:05:49 Systemnodes: 0, peers who asked us for Systemnode list: 0, peers we asked for Systemnode list: 0, entries in Systemnode list we asked for: 0
2018-12-12 03:05:49 Writing info to sncache.dat...
2018-12-12 03:05:49 Written info to sncache.dat 0ms
2018-12-12 03:05:49 Systemnodes: 0, peers who asked us for Systemnode list: 0, peers we asked for Systemnode list: 0, entries in Systemnode list we asked for: 0
2018-12-12 03:05:49 sncache.dat dump finished 0ms
2018-12-12 03:05:49 Verifying snpayments.dat format...
2018-12-12 03:05:49 Reading info from snpayments.dat...
2018-12-12 03:05:49 Loaded info from snpayments.dat 0ms
2018-12-12 03:05:49 Votes: 73, Blocks: 49
2018-12-12 03:05:49 Writing info to snpayments.dat...
2018-12-12 03:05:49 Written info to snpayments.dat 0ms
2018-12-12 03:05:49 Votes: 0, Blocks: 0
2018-12-12 03:05:49 snpayments.dat dump finished 2ms
2018-12-12 03:05:49 Verifying ixcache.dat format...
2018-12-12 03:05:49 Reading info from ixcache.dat...
2018-12-12 03:05:50 Loaded info from ixcache.dat 0ms
2018-12-12 03:05:50 Transaction lock requests: 0, Transaction locks: 0, Locked Inputs: 0, Transaction lock votes: 495
2018-12-12 03:05:50 Writing info to ixcache.dat...
2018-12-12 03:05:50 Written info to ixcache.dat 0ms
2018-12-12 03:05:50 Transaction lock requests: 0, Transaction locks: 0, Locked Inputs: 0, Transaction lock votes: 0
2018-12-12 03:05:50 ixcache.dat dump finished 12ms
2018-12-12 03:05:50 Shutdown: done
2018-12-12 03:06:01 Possible fixes
(Any comments on what you think might be responsible for the problem, if you have particular insight.)
/cc @finjosh