PoS testing: nodes go offline at random, can be hard/impossible to get back online
Summary
Nodes drop off the network at random. It can be very hard to get them back online.
Steps to reproduce
Start a handful of new nodes. Probably at least one will be offline within 8 hours.
Expected behavior
Nodes should not drop offline for no apparent reason.
Problematic behavior
Nodes drop offline for no apparent reason.
Crown-core environment info
Ubuntu 16.04
Crown-core application info
Crown version v0.12.7.1-2029222a pipeline 873
Relevant logs, dumps and/or screenshots
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code; it's very hard to read otherwise.)
Drop full logs & dumps here: https://nextcloud.crown.tech/nextcloud/s/znd8HiiAsRX3C6B
Possible fixes
Possibly related to the small network size, possibly related to the "lumpy" block production issue #288 (closed), possibly dues to nodes over-eagerly banning each other. When it happens with a wallet it sometimes helps to fire up a VPN and change the wallet IP address. This isn't practical for a node.
/cc @artem