[BUG] - Node shuts down after TrServerError

[linconvidal]

External

Area
Other Any other topic (Delegation, Ranking, ...).

Summary
I had a node running on preview for several days, but today I noticed it had shut down. It appears to be related to a connection issue, as it took less than 15 seconds from chain extension to shutdown. Restarting the node resolved the issue.

Steps to reproduce
I have not been able to reproduce the issue yet. I’m recording it here to bring attention to a potential problem.

Expected behavior
If there is a disconnection issue, the node should attempt to reconnect for more than just a few seconds before shutting down.

System info (please complete the following information):

• Running with Docker image ghcr.io/intersectmbo/cardano-node:9.1.1

Screenshots and attachments

cardano-node-1  | [0dd6344b:cardano.node.ChainDB:Notice:33] [2024-10-10 08:55:01.19 UTC] Chain extended, new tip: b36626d4ec63cdf6589e9b9c1c53de0a65f9d78770153c2c044e5f7b6c92625c at slot 61894501
cardano-node-1  | [0dd6344b:cardano.node.ChainDB:Notice:33] [2024-10-10 08:55:17.18 UTC] Chain extended, new tip: 686ecd4d9b906994f0a14886dfe40510b4fc94af7370654f3779f4d407a9699b at slot 61894517
cardano-node-1  | [0dd6344b:cardano.node.Mempool:Info:42765] [2024-10-10 08:55:29.50 UTC] fromList [("kind",String "TraceMempoolAddedTx"),("mempoolSize",Object (fromList [("bytes",Number 1295.0),("numTxs",Number 1.0)])),("tx",Object (fromList [("txid",String "b1b1ba29")]))]
cardano-node-1  | [0dd6344b:cardano.node.Server:Critical:59] [2024-10-10 08:55:33.31 UTC] TrServerError AsyncCancelled
cardano-node-1  | [0dd6344b:cardano.node.Server:Notice:51] [2024-10-10 08:55:33.31 UTC] TrServerStopped
cardano-node-1  | [0dd6344b:cardano.node.PeerSelection:Error:57] [2024-10-10 08:55:33.32 UTC] TraceOutboundGovernorCriticalFailure AsyncCancelled
cardano-node-1  | [0dd6344b:cardano.node.LocalServer:Critical:67] [2024-10-10 08:55:33.32 UTC] TrServerError AsyncCancelled
cardano-node-1  | [0dd6344b:cardano.node.LocalServer:Notice:62] [2024-10-10 08:55:33.32 UTC] TrServerStopped
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:62] [2024-10-10 08:55:33.32 UTC] TrShutdown
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:232] [2024-10-10 08:55:33.32 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 3, inboundConns = 3, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:240] [2024-10-10 08:55:33.32 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:242] [2024-10-10 08:55:33.32 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:1124] [2024-10-10 08:55:33.32 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:259] [2024-10-10 08:55:33.32 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:1121] [2024-10-10 08:55:33.32 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:51] [2024-10-10 08:55:33.33 UTC] TrShutdown
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48294] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 57, unidirectionalConns = 0, inboundConns = 3, outboundConns = 56})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49711] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 55, unidirectionalConns = 0, inboundConns = 3, outboundConns = 54})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:40716] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 50, unidirectionalConns = 0, inboundConns = 3, outboundConns = 49})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48175] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 49, unidirectionalConns = 0, inboundConns = 3, outboundConns = 48})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48328] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 47, unidirectionalConns = 0, inboundConns = 3, outboundConns = 46})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:45276] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 46, unidirectionalConns = 0, inboundConns = 3, outboundConns = 45})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47097] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 43, unidirectionalConns = 0, inboundConns = 3, outboundConns = 42})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48433] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 43, unidirectionalConns = 0, inboundConns = 3, outboundConns = 42})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48689] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 42, unidirectionalConns = 0, inboundConns = 3, outboundConns = 41})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47726] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 42, unidirectionalConns = 0, inboundConns = 3, outboundConns = 41})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49234] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 36, unidirectionalConns = 0, inboundConns = 3, outboundConns = 35})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48742] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 36, unidirectionalConns = 0, inboundConns = 3, outboundConns = 35})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:38526] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 35, unidirectionalConns = 0, inboundConns = 3, outboundConns = 34})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47319] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 34, unidirectionalConns = 0, inboundConns = 2, outboundConns = 34})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:43789] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 34, unidirectionalConns = 0, inboundConns = 2, outboundConns = 34})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:46331] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 34, unidirectionalConns = 0, inboundConns = 2, outboundConns = 34})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47698] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 29, unidirectionalConns = 0, inboundConns = 2, outboundConns = 29})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:38786] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 29, unidirectionalConns = 0, inboundConns = 2, outboundConns = 29})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49733] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 28, unidirectionalConns = 0, inboundConns = 2, outboundConns = 28})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49712] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 28, unidirectionalConns = 0, inboundConns = 2, outboundConns = 28})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49681] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 26, unidirectionalConns = 0, inboundConns = 2, outboundConns = 26})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49131] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 26, unidirectionalConns = 0, inboundConns = 2, outboundConns = 26})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49401] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 26, unidirectionalConns = 0, inboundConns = 2, outboundConns = 26})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47213] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 25, unidirectionalConns = 0, inboundConns = 2, outboundConns = 25})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:37792] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 25, unidirectionalConns = 0, inboundConns = 2, outboundConns = 25})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:45777] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 24, unidirectionalConns = 0, inboundConns = 2, outboundConns = 24})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47679] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 24, unidirectionalConns = 0, inboundConns = 2, outboundConns = 24})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:37940] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 22, unidirectionalConns = 0, inboundConns = 2, outboundConns = 22})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49753] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 21, unidirectionalConns = 0, inboundConns = 2, outboundConns = 21})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48729] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 21, unidirectionalConns = 0, inboundConns = 2, outboundConns = 21})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48467] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 21, unidirectionalConns = 0, inboundConns = 2, outboundConns = 21})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:40209] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 21, unidirectionalConns = 0, inboundConns = 2, outboundConns = 21})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:46268] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 21, unidirectionalConns = 0, inboundConns = 2, outboundConns = 21})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:45061] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 21, unidirectionalConns = 0, inboundConns = 2, outboundConns = 21})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48361] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 18, unidirectionalConns = 0, inboundConns = 2, outboundConns = 18})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49126] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 18, unidirectionalConns = 0, inboundConns = 2, outboundConns = 18})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47727] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 18, unidirectionalConns = 0, inboundConns = 2, outboundConns = 18})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48678] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 16, unidirectionalConns = 0, inboundConns = 1, outboundConns = 16})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:38191] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 16, unidirectionalConns = 0, inboundConns = 1, outboundConns = 16})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:45673] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 16, unidirectionalConns = 0, inboundConns = 1, outboundConns = 16})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:42027] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 10, unidirectionalConns = 0, inboundConns = 1, outboundConns = 10})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49434] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 9, unidirectionalConns = 0, inboundConns = 1, outboundConns = 9})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:44901] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 9, unidirectionalConns = 0, inboundConns = 1, outboundConns = 9})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:42978] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 8, unidirectionalConns = 0, inboundConns = 1, outboundConns = 8})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:45693] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 8, unidirectionalConns = 0, inboundConns = 1, outboundConns = 8})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49504] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 8, unidirectionalConns = 0, inboundConns = 1, outboundConns = 8})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49710] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 8, unidirectionalConns = 0, inboundConns = 1, outboundConns = 8})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:46636] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 6, unidirectionalConns = 0, inboundConns = 0, outboundConns = 6})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:42040] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 5, unidirectionalConns = 0, inboundConns = 0, outboundConns = 5})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48897] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 5, unidirectionalConns = 0, inboundConns = 0, outboundConns = 5})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:42821] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 5, unidirectionalConns = 0, inboundConns = 0, outboundConns = 5})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48419] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 3, unidirectionalConns = 0, inboundConns = 0, outboundConns = 3})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:42783] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 3, unidirectionalConns = 0, inboundConns = 0, outboundConns = 3})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49687] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 2, unidirectionalConns = 0, inboundConns = 0, outboundConns = 2})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:43928] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49141] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48677] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49171] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:40635] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49474] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47370] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.DiffusionInitializationTracer:Critical:5] [2024-10-10 08:55:33.34 UTC] DiffusionErrored thread killed
cardano-node-1  | [0dd6344b:cardano.node.ChainDB:Info:5] [2024-10-10 08:55:33.35 UTC] Closed Immutable DB.
cardano-node-1  | [0dd6344b:cardano.node.ChainDB:Info:5] [2024-10-10 08:55:33.35 UTC] Closed Volatile DB.
cardano-node-1  | [0dd6344b:cardano.node.ChainDB:Info:5] [2024-10-10 08:55:33.35 UTC] Closed db with immutable tip at d7941d06ad23506f4e858f1a4300996763c8ef48e4368bcd048389a6f9520d01 at slot 61882724 and tip 686ecd4d9b906994f0a14886dfe40510b4fc94af7370654f3779f4d407a9699b at slot 61894517
cardano-node-1  | 
cardano-node-1  | Shutting down..
cardano-node-1  | cardano-node: thread killed

[carbolymer]

cardano-node-1 | [0dd6344b:cardano.node.DiffusionInitializationTracer:Critical:5] [2024-10-10 08:55:33.34 UTC] DiffusionErrored thread killed

Have you checked if it was not an operating system fault, in system logs for example? Have you checked if you have not run out of memory?

[coot]

The relevant section of the log doesn't show where the problem originated from:

cardano-node-1  | [0dd6344b:cardano.node.Server:Critical:59] [2024-10-10 08:55:33.31 UTC] TrServerError AsyncCancelled
cardano-node-1  | [0dd6344b:cardano.node.Server:Notice:51] [2024-10-10 08:55:33.31 UTC] TrServerStopped
cardano-node-1  | [0dd6344b:cardano.node.PeerSelection:Error:57] [2024-10-10 08:55:33.32 UTC] TraceOutboundGovernorCriticalFailure AsyncCancelled
cardano-node-1  | [0dd6344b:cardano.node.LocalServer:Critical:67] [2024-10-10 08:55:33.32 UTC] TrServerError AsyncCancelled
cardano-node-1  | [0dd6344b:cardano.node.LocalServer:Notice:62] [2024-10-10 08:55:33.32 UTC] TrServerStopped
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:62] [2024-10-10 08:55:33.32 UTC] TrShutdown

asynchronous exceptions mean that some other thread decided to kill the server. Could you share full logs?