Panics at update_registration

[majecty]

A node which running Tendermint consensus panicked with below error.
Two of P2P workers panicked at update_registration function.

====================

stack backtrace:
   0:     0x5568c17977a5 - backtrace::backtrace::libunwind::trace::h4f79420391b53c5e
                        at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.7/src/backtrace/libunwind.rs:53
                         - backtrace::backtrace::trace::h9b33075d5557de4b
                        at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.7/src/backtrace/mod.rs:42
   1:     0x5568c1791f3e - backtrace::capture::Backtrace::new_unresolved::hec68c6b18a3d0ccb
                        at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.7/src/capture.rs:88
   2:     0x5568c1791e9c - backtrace::capture::Backtrace::new::h52fad09914409242
                        at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.7/src/capture.rs:63
   3:     0x5568c047af51 - panic_hook::panic_hook::he90a460657415dd4
                        at util/panic_hook/src/lib.rs:53
   4:     0x5568c047b537 - core::ops::function::Fn::call::haf5c3a62c69defcf
                        at libcore/ops/function.rs:78
   5:     0x5568c1d9d553 - std::panicking::rust_panic_with_hook::h0e12cb2fc86d00fa
                        at libstd/panicking.rs:481
   6:     0x5568c1d9d0b9 - std::panicking::continue_panic_fmt::h141671b29fe0e27d
                        at libstd/panicking.rs:391
   7:     0x5568c1d9cfb5 - rust_begin_unwind
                        at libstd/panicking.rs:326
   8:     0x5568c1df2e0b - core::panicking::panic_fmt::h429a06507aba9228
                        at libcore/panicking.rs:77
   9:     0x5568c1445323 - core::result::unwrap_failed::h37cc62584b5577fe
                        at libcore/macros.rs:26
  10:     0x5568c144036a - <core::result::Result<T, E>>::unwrap::h8b450c305bf15e95
                        at libcore/result.rs:808
  11:     0x5568c14c1acb - <codechain_io::service::IoContext<Message>>::update_registration::h3d0818a083d2eed6
                        at /code/kodebox/codechain-1/util/io/src/service.rs:155
  12:     0x5568c14293be - <codechain_network::p2p::handler::Handler as codechain_io::IoHandler<codechain_network::p2p::handler::Message>>::stream_writable::{{closure}}::h50bfaed8df87b0a4
                        at network/src/p2p/handler.rs:492
  13:     0x5568c14356e2 - <codechain_finally::Finally<F> as core::ops::drop::Drop>::drop::h7c580c0ae48f53f2
                        at /code/kodebox/codechain-1/util/finally/src/lib.rs:30
  14:     0x5568c14302bd - core::ptr::drop_in_place::hf56d6b9f533d4b3f
                        at libcore/ptr.rs:59
  15:     0x5568c14f5e61 - <codechain_network::p2p::handler::Handler as codechain_io::IoHandler<codechain_network::p2p::handler::Message>>::stream_writable::h1d3111d8ba1d219a
                        at network/src/p2p/handler.rs:495
  16:     0x5568c145c1ec - codechain_io::worker::Worker::do_work::h16d7a951b03e3674
                        at /code/kodebox/codechain-1/util/io/src/worker.rs:128
  17:     0x5568c145efa6 - codechain_io::worker::Worker::work_loop::h118f72cd8dc4169d
                        at /code/kodebox/codechain-1/util/io/src/worker.rs:111
  18:     0x5568c145bb0d - codechain_io::worker::Worker::new::{{closure}}::h639e167213ef7971
                        at /code/kodebox/codechain-1/util/io/src/worker.rs:85
  19:     0x5568c1502dd1 - std::sys_common::backtrace::__rust_begin_short_backtrace::hf9d296eb63328325
                        at libstd/sys_common/backtrace.rs:136
  20:     0x5568c14fceb3 - std::thread::Builder::spawn::{{closure}}::{{closure}}::hadef334dc2eb4491
                        at libstd/thread/mod.rs:409
  21:     0x5568c150b1a3 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h4c02341ba5f06d29
                        at libstd/panic.rs:313
  22:     0x5568c150df99 - std::panicking::try::do_call::he291a46897ab8ddf
                        at libstd/panicking.rs:310
  23:     0x5568c1dc6a89 - __rust_maybe_catch_panic
                        at libpanic_unwind/lib.rs:103
  24:     0x5568c150da80 - std::panicking::try::h97aae5a367324163
                        at libstd/panicking.rs:289
  25:     0x5568c150b263 - std::panic::catch_unwind::h181daed8d16c2d81
                        at libstd/panic.rs:392
  26:     0x5568c14fc568 - std::thread::Builder::spawn::{{closure}}::h33c1cb2293bb65af
                        at libstd/thread/mod.rs:408
  27:     0x5568c14fd0dd - <F as alloc::boxed::FnBox<A>>::call_box::hfff9797e3957f3e6
                        at liballoc/boxed.rs:646
  28:     0x5568c1db5dda - <alloc::boxed::Box<(dyn alloc::boxed::FnBox<A, Output$u3d$R$GT$$u20$$u2b$$u20$$u27$a$RP$$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hd022a0500f5eee13
                        at liballoc/boxed.rs:656
                         - std::sys_common::thread::start_thread::hb77935aee02382da
                        at libstd/sys_common/thread.rs:24
  29:     0x5568c1d9dcb5 - std::sys::unix::thread::Thread::new::thread_start::hd61f7429b3de1d75
                        at libstd/sys/unix/thread.rs:90
  30:     0x7ffb2bb596da - start_thread
  31:     0x7ffb2b66a88e - __clone
  32:                0x0 - <unknown>

Thread 'P2P Worker #1' panicked at 'called `Result::unwrap()` on an `Err` value: Mio(Custom { kind: ConnectionAborted, error: StringError("Network IO notification error") })', libcore/result.rs:1009

This is a bug. Please report it at:

    https://github.com/CodeChain-io/codechain/issues/new

stack backtrace:
   0:     0x5568c17977a5 - backtrace::backtrace::libunwind::trace::h4f79420391b53c5e
                        at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.7/src/backtrace/libunwind.rs:53
                         - backtrace::backtrace::trace::h9b33075d5557de4b
                        at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.7/src/backtrace/mod.rs:42
   1:     0x5568c1791f3e - backtrace::capture::Backtrace::new_unresolved::hec68c6b18a3d0ccb
                        at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.7/src/capture.rs:88
   2:     0x5568c1791e9c - backtrace::capture::Backtrace::new::h52fad09914409242
                        at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.7/src/capture.rs:63
   3:     0x5568c047af51 - panic_hook::panic_hook::he90a460657415dd4
                        at util/panic_hook/src/lib.rs:53
   4:     0x5568c047b537 - core::ops::function::Fn::call::haf5c3a62c69defcf
                        at libcore/ops/function.rs:78
   5:     0x5568c1d9d553 - std::panicking::rust_panic_with_hook::h0e12cb2fc86d00fa
                        at libstd/panicking.rs:481
   6:     0x5568c1d9d0b9 - std::panicking::continue_panic_fmt::h141671b29fe0e27d
                        at libstd/panicking.rs:391
   7:     0x5568c1d9cfb5 - rust_begin_unwind
                        at libstd/panicking.rs:326
   8:     0x5568c1df2e0b - core::panicking::panic_fmt::h429a06507aba9228
                        at libcore/panicking.rs:77
   9:     0x5568c1445323 - core::result::unwrap_failed::h37cc62584b5577fe
                        at libcore/macros.rs:26
  10:     0x5568c144036a - <core::result::Result<T, E>>::unwrap::h8b450c305bf15e95
                        at libcore/result.rs:808
  11:     0x5568c14c1acb - <codechain_io::service::IoContext<Message>>::update_registration::h3d0818a083d2eed6
                        at /code/kodebox/codechain-1/util/io/src/service.rs:155
  12:     0x5568c14293be - <codechain_network::p2p::handler::Handler as codechain_io::IoHandler<codechain_network::p2p::handler::Message>>::stream_writable::{{closure}}::h50bfaed8df87b0a4
                        at network/src/p2p/handler.rs:492
  13:     0x5568c14356e2 - <codechain_finally::Finally<F> as core::ops::drop::Drop>::drop::h7c580c0ae48f53f2
                        at /code/kodebox/codechain-1/util/finally/src/lib.rs:30
  14:     0x5568c14302bd - core::ptr::drop_in_place::hf56d6b9f533d4b3f
                        at libcore/ptr.rs:59
  15:     0x5568c14f5e61 - <codechain_network::p2p::handler::Handler as codechain_io::IoHandler<codechain_network::p2p::handler::Message>>::stream_writable::h1d3111d8ba1d219a
                        at network/src/p2p/handler.rs:495
  16:     0x5568c145c1ec - codechain_io::worker::Worker::do_work::h16d7a951b03e3674
                        at /code/kodebox/codechain-1/util/io/src/worker.rs:128
  17:     0x5568c145efa6 - codechain_io::worker::Worker::work_loop::h118f72cd8dc4169d
                        at /code/kodebox/codechain-1/util/io/src/worker.rs:111
  18:     0x5568c145bb0d - codechain_io::worker::Worker::new::{{closure}}::h639e167213ef7971
                        at /code/kodebox/codechain-1/util/io/src/worker.rs:85
  19:     0x5568c1502dd1 - std::sys_common::backtrace::__rust_begin_short_backtrace::hf9d296eb63328325
                        at libstd/sys_common/backtrace.rs:136
  20:     0x5568c14fceb3 - std::thread::Builder::spawn::{{closure}}::{{closure}}::hadef334dc2eb4491
                        at libstd/thread/mod.rs:409
  21:     0x5568c150b1a3 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h4c02341ba5f06d29
                        at libstd/panic.rs:313
  22:     0x5568c150df99 - std::panicking::try::do_call::he291a46897ab8ddf
                        at libstd/panicking.rs:310
  23:     0x5568c1dc6a89 - __rust_maybe_catch_panic
                        at libpanic_unwind/lib.rs:103
  24:     0x5568c150da80 - std::panicking::try::h97aae5a367324163
                        at libstd/panicking.rs:289
  25:     0x5568c150b263 - std::panic::catch_unwind::h181daed8d16c2d81
                        at libstd/panic.rs:392
  26:     0x5568c14fc568 - std::thread::Builder::spawn::{{closure}}::h33c1cb2293bb65af
                        at libstd/thread/mod.rs:408
  27:     0x5568c14fd0dd - <F as alloc::boxed::FnBox<A>>::call_box::hfff9797e3957f3e6
                        at liballoc/boxed.rs:646
  28:     0x5568c1db5dda - <alloc::boxed::Box<(dyn alloc::boxed::FnBox<A, Output$u3d$R$GT$$u20$$u2b$$u20$$u27$a$RP$$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hd022a0500f5eee13
                        at liballoc/boxed.rs:656
                         - std::sys_common::thread::start_thread::hb77935aee02382da
                        at libstd/sys_common/thread.rs:24
  29:     0x5568c1d9dcb5 - std::sys::unix::thread::Thread::new::thread_start::hd61f7429b3de1d75
                        at libstd/sys/unix/thread.rs:90
  30:     0x7ffb2bb596da - start_thread
  31:     0x7ffb2b66a88e - __clone
  32:                0x0 - <unknown>

Thread 'P2P Worker #2' panicked at 'called `Result::unwrap()` on an `Err` value: Mio(Custom { kind: ConnectionAborted, error: StringError("Network IO notification error") })', libcore/result.rs:1009

This is a bug. Please report it at:

    https://github.com/CodeChain-io/codechain/issues/new

[majecty]

This happens at 84a1197

====================

stack backtrace:
   0:     0x56191aa0c3a6 - backtrace::backtrace::libunwind::trace::h6f69f7a8523b3b48
                        at /home/majecty/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.7/src/backtrace/libunwind.rs:53
                         - backtrace::backtrace::trace::hfcb48dcb3ac891e8
                        at /home/majecty/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.7/src/backtrace/mod.rs:42
   1:     0x56191aa06953 - backtrace::capture::Backtrace::new_unresolved::h3e5684a38718154b
                        at /home/majecty/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.7/src/capture.rs:88
   2:     0x56191aa068ad - backtrace::capture::Backtrace::new::h448ec18ea4124027
                        at /home/majecty/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.7/src/capture.rs:63
   3:     0x56191992539d - panic_hook::panic_hook::ha4aecdc1727731f8
                        at util/panic_hook/src/lib.rs:53
   4:     0x561919925d47 - core::ops::function::Fn::call::h50bcd1f0a1621bec
                        at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/ops/function.rs:78
   5:     0x56191b3d37d9 - std::panicking::rust_panic_with_hook::h8cbdfe43764887be
                        at src/libstd/panicking.rs:495
   6:     0x56191b3d3281 - std::panicking::continue_panic_fmt::h3d3c5a833c00a5e1
                        at src/libstd/panicking.rs:398
   7:     0x56191b3d3165 - rust_begin_unwind
                        at src/libstd/panicking.rs:325
   8:     0x56191b3eeddc - core::panicking::panic_fmt::h4d67173bc68f6d5a
                        at src/libcore/panicking.rs:95
   9:     0x56191ab39a80 - core::result::unwrap_failed::h31d96726ae460c7c
                        at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/macros.rs:26
  10:     0x56191ab3489a - <core::result::Result<T, E>>::unwrap::hf2bd88381cc06101
                        at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/result.rs:808
  11:     0x56191aa629af - <codechain_io::service::IoContext<Message>>::update_registration::hce8df400405635de
                        at /home/majecty/code/kodebox/codechain/util/io/src/service.rs:137
  12:     0x56191ab5324c - <codechain_network::p2p::handler::Handler as codechain_io::IoHandler<codechain_network::p2p::handler::Message>>::message::hdf202b4b64dea649
                        at network/src/p2p/handler.rs:396
  13:     0x56191aa47dec - codechain_io::worker::Worker::do_work::ha7020aeff310c5f5
                        at /home/majecty/code/kodebox/codechain/util/io/src/worker.rs:142
  14:     0x56191aa48c74 - codechain_io::worker::Worker::work_loop::hcb5a3916cfbe5618
                        at /home/majecty/code/kodebox/codechain/util/io/src/worker.rs:110
  15:     0x56191aa45392 - codechain_io::worker::Worker::new::{{closure}}::hf8dd10cf19a2954e
                        at /home/majecty/code/kodebox/codechain/util/io/src/worker.rs:84
  16:     0x56191ab55ad0 - std::sys_common::backtrace::__rust_begin_short_backtrace::h161f735aea7350e5
                        at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/sys_common/backtrace.rs:136
  17:     0x56191aa74372 - std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::h8f961ef5f7a33a05
                        at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/thread/mod.rs:477
  18:     0x56191aaaa882 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h0d49bbbc0ba9b155
                        at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panic.rs:319
  19:     0x56191ab63dd6 - std::panicking::try::do_call::hc33962e6ae0d810d
                        at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panicking.rs:310
  20:     0x56191b3e8569 - __rust_maybe_catch_panic
                        at src/libpanic_unwind/lib.rs:102
  21:     0x56191ab637ff - std::panicking::try::h80fe847628a380e0
                        at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panicking.rs:289
  22:     0x56191aaaaa72 - std::panic::catch_unwind::hac356190652458ce
                        at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panic.rs:398
  23:     0x56191aa73a1d - std::thread::Builder::spawn_unchecked::{{closure}}::hbf8a0579c78392f1
                        at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/thread/mod.rs:476
  24:     0x56191aa74737 - <F as alloc::boxed::FnBox<A>>::call_box::h8d2cc887ac93e8d3
                        at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/liballoc/boxed.rs:673
  25:     0x56191b3dc14d - <alloc::boxed::Box<(dyn alloc::boxed::FnBox<A, Output$u3d$R$GT$$u20$$u2b$$u20$$u27$a$RP$$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hece536cf07b94f8d
                        at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/liballoc/boxed.rs:683
                         - std::sys_common::thread::start_thread::h9605a7df0f911844
                        at src/libstd/sys_common/thread.rs:24
                         - std::sys::unix::thread::Thread::new::thread_start::hca8e72c41fa9d291
                        at src/libstd/sys/unix/thread.rs:90
  26:     0x7f23cd3266da - start_thread
  27:     0x7f23cce3788e - __clone
  28:                0x0 - <unknown>

Thread 'P2P Worker #3' panicked at 'called `Result::unwrap()` on an `Err` value: Mio(Custom { kind: ConnectionAborted, error: StringError("Network IO notification error") })', src/libcore/result.rs:1009

This is a bug. Please report it at:

    https://github.com/CodeChain-io/codechain/issues/new

thread panicked while processing panic. aborting.

[majecty]

2019-02-02 10:25:23 UTC corgi-us-west-1-2

[sgkim126]

I added a commit to log more information b744020

[sgkim126]

#1204 may fix it. Please reopen it if it happens again.