Skip to content

GitLab

Closed
Created by Mark Brooker@walkjiveflyMaintainer

PoS testing: CreateCoinStake -- Couldn't find recent payment blocks for MN

Summary

MN seized up. Doesn't respond to crown-cli commands. debug.log full of repeated

CreateCoinStake -- Couldn't find recent payment blocks for MN

messages which started about a minute after the node was started and immediately after it was enabled.

Steps to reproduce

Unknown

Expected behavior

Not a log full of error messages

Problematic behavior

Log full of error messages

Crown-core environment info

Ubuntu 16.04

Crown-core application info

Crown version v0.12.7.3-2cf48571 pipeline 896 job 5181

Relevant logs, dumps and/or screenshots

...
2019-01-31 15:00:52 CreateCoinStake -- Couldn't find CMasternode object for active masternode
2019-01-31 15:00:52 mnb - Got NEW Masternode entry - 44261255ee5c564d9fd965a2c7767f9643298f86dce6e26fedf907b844136957 - 164.132.151.93:19340 - CTxIn(COutPoint(5b33dbec20784166f310f6059e719fe4d2622c71cbee183b4fe739eb034192b4, 1), scriptSig=) - 1548946852
2019-01-31 15:00:52 CActiveMasternode::EnableHotColdMasterNode() - Enabled! You may shut down the cold daemon.
2019-01-31 15:00:52 CheckInputsAndAdd: Verified pubkey2 signover for staking, added to activemasternode
2019-01-31 15:00:52 CMasternodeSync:ProcessMessage - ssc - got inventory count 3 156
2019-01-31 15:00:53 CSystemnodeSync:ProcessMessage - snssc - got inventory count 3 720
2019-01-31 15:00:53 CreateCoinStake -- Couldn't find recent payment blocks for MN
2019-01-31 15:00:54 CreateCoinStake -- Couldn't find recent payment blocks for MN
2019-01-31 15:00:55 CreateCoinStake -- Couldn't find recent payment blocks for MN
...
2019-02-01 17:23:17 CreateCoinStake -- Couldn't find recent payment blocks for MN
2019-02-01 17:23:18 CreateCoinStake -- Couldn't find recent payment blocks for MN
2019-02-01 17:23:19 CreateCoinStake -- Couldn't find recent payment blocks for MN

Took a look with gdb:

crown@Crown-Testnet:~/.crown/testnet3$ sudo gdb --pid=`pidof crownd`
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 19287
[New LWP 19290]
[New LWP 19291]
[New LWP 19292]
[New LWP 19293]
[New LWP 19294]
[New LWP 19295]
[New LWP 19296]
[New LWP 19297]
[New LWP 19298]
[New LWP 19299]
[New LWP 19300]
[New LWP 19301]
[New LWP 19303]
[New LWP 19305]
[New LWP 19306]
[New LWP 19307]
[New LWP 19308]
[New LWP 19309]
[New LWP 19310]
[New LWP 19311]
[New LWP 19322]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185	../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
(gdb) info threds
Undefined info command: "threds".  Try "help info".
(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 0x7f51796a5740 (LWP 19287) "crownd" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  2    Thread 0x7f51781e6700 (LWP 19290) "crownd" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  3    Thread 0x7f51779e5700 (LWP 19291) "crown-scriptch" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  4    Thread 0x7f51771e4700 (LWP 19292) "crown-scriptch" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  5    Thread 0x7f51769e3700 (LWP 19293) "crown-scriptch" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  6    Thread 0x7f51761e2700 (LWP 19294) "crown-scriptch" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  7    Thread 0x7f51759e1700 (LWP 19295) "crown-scriptch" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  8    Thread 0x7f51751e0700 (LWP 19296) "crown-scriptch" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  9    Thread 0x7f51749df700 (LWP 19297) "crown-scriptch" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  10   Thread 0x7f51741de700 (LWP 19298) "crownd" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  11   Thread 0x7f51739dd700 (LWP 19299) "crownd" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 
  12   Thread 0x7f51731dc700 (LWP 19300) "crownd" 0x00007f51782eea13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
  13   Thread 0x7f51729db700 (LWP 19301) "crownd" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  14   Thread 0x7f516bfff700 (LWP 19303) "crown-legacysig" pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  15   Thread 0x7f516affd700 (LWP 19305) "crown-net" 0x00007f51782e45d3 in select () at ../sysdeps/unix/syscall-template.S:84
  16   Thread 0x7f516a7fc700 (LWP 19306) "crown-addcon" pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  17   Thread 0x7f5169ffb700 (LWP 19307) "crown-opencon" 0x00007f51782e45d3 in select () at ../sysdeps/unix/syscall-template.S:84
  18   Thread 0x7f51697fa700 (LWP 19308) "crown-msghand" pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  19   Thread 0x7f5168ff9700 (LWP 19309) "crown-dumpaddr" pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  20   Thread 0x7f514bfff700 (LWP 19310) "crown-miner" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  21   Thread 0x7f514b7fe700 (LWP 19311) "crown-wallet" pthread_cond_timedwait@@---Type <return> to continue, or q <return> to quit---
GLIBC_2.3.2 ()
    at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  22   Thread 0x7f514affd700 (LWP 19322) "crown-msghand" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
(gdb) quit

Uploaded 20190201_walkjivefly_MN_PMN01_seize.log to https://nextcloud.crown.tech/nextcloud/s/znd8HiiAsRX3C6B

Possible fixes

(Any comments on what you think might be responsible for the problem, if you have particular insight.)

/cc @artem

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information