Skip to content

GitLab

Open
Created by Mark Brooker@walkjiveflyMaintainer

Node "hangs" following socket sending timeout: 1201s

MN running Crown Core version v0.21.0.0-3b900b87 (release build) hung after running for about 16 minutes. 20 minutes later it timed out it's peers and had to be killed.

gdb showed

(gdb) bt
#0  0x00007f49d14bcc1d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x00005562f4b63a9e in UninterruptibleSleep(std::chrono::duration<long, std::ratio<1l, 1000000l> > const&) ()
#2  0x00005562f458eb43 in AppInit(int, char**) ()
#3  0x00005562f457fb3f in main ()
(gdb) info thread
  Id   Target Id         Frame 
* 1    Thread 0x7f49d18de740 (LWP 15137) "crownd" 0x00007f49d14bcc1d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
  2    Thread 0x7f49ce35e700 (LWP 15139) "b-scriptch.0" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  3    Thread 0x7f49cdb5d700 (LWP 15140) "b-scriptch.1" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  4    Thread 0x7f49c7fff700 (LWP 15141) "b-scriptch.2" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  5    Thread 0x7f49cd35c700 (LWP 15142) "b-crown-mnodesy" __lll_lock_wait ()
    at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  6    Thread 0x7f49ccb5b700 (LWP 15143) "b-http" 0x00007f49d0740b13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
  7    Thread 0x7f49c77fe700 (LWP 15144) "b-httpworker.0" __lll_lock_wait ()
    at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  8    Thread 0x7f49c6ffd700 (LWP 15145) "b-httpworker.1" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  9    Thread 0x7f49c67fc700 (LWP 15146) "b-httpworker.2" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  10   Thread 0x7f49c5ffb700 (LWP 15147) "b-httpworker.3" __lll_lock_wait ()
    at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  11   Thread 0x7f49c57fa700 (LWP 15148) "crownd" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  12   Thread 0x7f497574f700 (LWP 15160) "b-torcontrol" 0x00007f49d0740b13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
  13   Thread 0x7f496ffff700 (LWP 15161) "b-net" 0x00007f49d073484d in poll ()
    at ../sysdeps/unix/syscall-template.S:84
  14   Thread 0x7f4967fff700 (LWP 15163) "b-addcon" __lll_lock_wait ()
    at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  15   Thread 0x7f496effd700 (LWP 15164) "b-mncon" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  16   Thread 0x7f496e7fc700 (LWP 15165) "b-opencon" __lll_lock_wait ()
    at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  17   Thread 0x7f496dffb700 (LWP 15166) "b-msghand" __lll_lock_wait ()
    at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135

Uploaded debug.log to https://storage.crownplatform.com/f/23104

core file is still on xmn03 in ~crown/.crown/issue-002.core (and debug.log is in ~crown/.crown/issue-002.log)

tail of the debug.log showed

2021-06-23T12:53:11Z NodeMinter: Attempting to stake..
2021-06-23T12:53:11Z [default wallet] keypool reserve 1001
2021-06-23T12:53:11Z [default wallet] keypool return 1001
2021-06-23T12:53:16Z GetRecentStakePointer -- Couldn't find last paid block
2021-06-23T12:53:16Z CreateCoinStake -- Couldn't find recent payment blocks for MN
2021-06-23T12:53:16Z NodeMinter: Stake not found..
2021-06-23T13:13:12Z socket sending timeout: 1201s
2021-06-23T13:13:12Z socket sending timeout: 1201s
2021-06-23T13:13:17Z socket sending timeout: 1201s
2021-06-23T13:13:17Z socket sending timeout: 1201s
2021-06-23T13:13:17Z socket sending timeout: 1201s
2021-06-23T13:13:17Z socket sending timeout: 1201s
2021-06-23T13:13:17Z socket sending timeout: 1201s
2021-06-23T13:13:17Z socket sending timeout: 1201s
2021-06-23T13:13:17Z socket sending timeout: 1201s
2021-06-23T13:13:17Z socket sending timeout: 1201s
Edited by Mark Brooker